Surely with the tax/interest i cannot be allowed to have a bank account in each country?
Of course you can - you just have to declare the interest to the Polish taxman. Or don't - it's entirely up to you. At the end of the year when you fill in the PIT form, you just add the interest received from the UK bank account. I think you can also tell the UK bank that you aren't resident for tax purposes (not in the UK for more than 185 days a year) and so you'll receive interest gross, not net.
UK banks don't particularly care that much if you aren't resident in the UK - as long as you maintain an address in the UK (parents/uncle/aunt/cousins/whatever) - that's all they care about.
But you will need a Polish bank account to satisfy their demands at the Foreigners Office. mBank is probably the easiest as a Brit, because their systems are actually set up to cope with foreigners without residence permits.