he's told me I can't leave with our 3 year old daughter without his written permission? Is this true?
In theory you can't leave Poland with the child without his permission. In theory you could apply to the court for permission to travel with the kid if you want to; the fact that he has abandoned his family and has virtually no contact with his daughter is not going to go in his favour. However, the Polish family court system is entirely unfit for purpose and can be an absolute nightmare. I'd strongly suggest steering well clear of it.
I would only be able to stay here with a lot of financial support from him which I really don't see happening.
The Polish family court would most likely rule in your favour (i.e. award you custody and about 800 to 1,000zl a month in maintenance).