I also have difficulty with some of the vowels in Polish. The "ę," for instance, always throws me for a loop. I will probably always stumble over that vowel. For example, when I try to say "Piękny," it always comes out like I'm saying "pinknay" or something like that. The ł sound is hard for me, too. I could go on and on.
This is the key. Don't exaggerate your own or your neighbours' expectations; they'll be thrilled (especially as they're not in Poland, this always adds an extra dimension) with a simple greeting - cześć/witam/dzień dobry - and you can take it from there. In fact, it's your neighbours who'll give you the best exposure to the language on an everyday level.
Yeah, they seem like nice people. I am just afraid that they might not understand it when I try to talk to them in their own language or, even worse, think I'm crazy. Oh well, I just need to get confidence.