It often comes across as being rude when you don't use Pan / Pani when talking to people you don't know or people you only interact with on a professional level.
'Pan' is a bit like English term 'Sir'
'Pani' means 'Ma'am'
Most of the time there is no need to use those with friends and friends of your friends that you hang out with.
when speaking to more than one person (e.g. a couple). It gets even more complicated when speaking to professional people (Pan doktor/profesor
etc) but don't worry too much about that for now.
The best rule to use is that it's better to be a little too formal, rather than risk offending someone by being too informal. However, it's also true to say that most people will take into account that you aren't Polish, or you don't speak the language fluently.