I believe that once you come to live in Poland permanently, (i.e. more than 186 days a year) and have your base in Poland (i.e. prime residency)
That is correct but in case of employees, who carry out work in Poland, there's no doubt where their "base" is, while in case of let's say German or British retired people, who rent/own some property in Spain or Portugal, where is their "base" ? Who can tell If they spend 200 days a year in Germany and 165 in Portugal or the other way around.
Here it is in case of PL/Swiss -> przepisy.gofin.pl/przepisy,4,15,82,178,,,konwencja-miedzy-rzeczapospolita-polska-a.html -> Art. 4, point 2 c)