In all honesty, it's pretty unlikely that you'll meet trouble. I would certainly recommend to keep your wits about you, don't go to the bad parts of town and don't draw attention to yourself in situations where knuckleheads might notice - but apart from that, I'm pretty sure that you'll be fine :) That advice actually applies to anyone really - skin colour doesn't really come into it.
Do be prepared for some stares - but this isn't always rude or offensive, as it can simply be curiosity. You might get some muttered comments - but to be honest, this applies to all tourists, not just ones of a different skin colour. I take particular pride in winding up people on the bus to my school - there's one guy who we always see that clearly hates English speakers, yet he's far too polite to actually do anything but move seat.
Where about in Poland do you intend to go?