When I spend time with him things are great and his mom likes me
If his mom likes you, you're a step ahead. Some moms aren't so nice. I wouldn't worry about the rest of the family speaking in Polish around you. You're not dating them. All that matters is that your boyfriend likes you.