And we are not sure if I can travel to Poland with this residency card, issued in the UK, without visa.
No, you can't I'm afraid. You'll need to get a visa issued by Poland (or other Schengen country). The visa is free, and will be processed quickly, but it's still required.
All details are here.