As has been pointed out several times, Polak is a Polish man, but it is not enough to say "a person hailing from Poland" because that exludes Polki, who are of course the women hailing from Poland. But that's not at issue here, I know.
I agree with the comparison ArturSzastak made with not using the N word to a black person unless you're black. It's unfortunate that a word should be reserved as appropriate to only certain people, but that is the way it is.
I also don't think it's the same thing for a Ukrainian or Russian or any other Slav to use the term Polak because they are coming from a shared linguistic background, and it is safe to assume that their intentions are neutral. English speakers cannot claim this.
I am Polka, not Polak. The only time anyone ever calls me Polak it is not out of curiousity about my heritage, but in order to reference the "Polack joke" mentality. I find it offensive coming from someone who doesn't know what it means and isn't using it correctly.
It's perfectly fine to mix languages, most multilinguals do so, but if you don't know Polish, this should not be the one Polish word you insert into conversation.
Bottom line, if there's any controversy about a word, I would avoid it just in case. No one will get offended because you DON'T use the word Polak, but since some people might, I'd stay away from it. Unless, of course, you're speaking in Polish.