And we are not sure if I can travel to Poland with this residency card, issued in the UK, without visa.
No, you can't I'm afraid. You'll need to get a visa issued by Poland (or other Schengen country). The visa is free, and will be processed quickly, but it's still required.
europa.eu/youreurope/citizens/travel/entry-exit/non-eu-family/index_en.htm
All details are here.