It was almost impossible. People would come back to occupied Poland with great news about freedom and democracy that exists somewhere over the rainbow. They would also bring information about real face of communist occupation to the West.
Usually people who held passport were strong communists, communist party members or their families etc.
My family had passports, I had grandpa in the party. Not good news but he was good man anyway.