Law /
EU citizen residency certificate to live in Poland [34]
You must actually cut it yourself, fold it in two and keep it in your wallet.
There's no obligation to carry it - as I understand it, it isn't proof of identity anyway, but rather "confirmation of the right of residence as a European Union national" and thus useless for any sort of identification purposes.
Can you actually benefit from Polish social security and medical care if you don't have a PESEL?
Yes, of course. Medical care depends on ZUS contributions, nothing more - likewise, social security depends on your record at the Urzad Skarbowy. It's possible that some people might argue, but having a PESEL isn't a requirement for non-Polish citizens.
One thing - no PESEL means no driving record with the police and thus no points. It's why it's a good idea not to hold a Polish licence - most police aren't tuned into the fact that we foreigners can have a PESEL number and it's best to not let them know that ;)
(not sure if the apply after 4 days arrival still applies)
If they ask you, tell them any old rubbish about how you crossed a few days ago. They've no way of checking whatsoever.
Hard to believ that I could just buy a piece of paper that would solve this problem.
That's because it won't - as a non-EU citizen, you aren't entitled to the EU residence permit. You've already got permanent residence, but you can only exercise your treaty rights if your wife accompanies you - you can't go elsewhere without her to work. But if you have permanent residence, aren't you entitled to citizenship - you can apply for it after as little as 3 years of marriage.
My wife is Polish and it seems like the easiest thing for us to do, would be to have her go live in the 'other' country first, and then bring me over.
In theory, yes. But remember, there are still quite a few restrictions for Poles - the UK requires the WRS registration, Germany and Austria still require work permits, and it's not just as simple as moving elsewhere and working legally. You'll also have to deal with all the bureaucracy in other countries, including applying for a residence permit - and believe me, Poland is quite easy when it comes to this kind of thing.
but will be arriving/staying using my british EU passport
You don't actually need the EU residence permit at all - if you don't have a reason for obtaining it, then you might as well not bother.
residency certificate from your native country
Do other countries have this insane system of registration, too?