we end up going to Poland what happens to me?
Who said you will be allowed in Poland? As a non-EU citizen you won't have any rights. Maybe Poland would give you a one month holiday visa, nobody knows.
You would have to learn Polish if you wanted to live and work in Poland.
You would have to, as an unskilled builder he would return along with several hundred thousand other Poles and there will be no work. Either go to another EU country (you would have to learn that language) or sit at home in his village drinking vodka from your 500pln a child.
Have another baby, 1000pln vodka money and he is sorted.