we can take an average and say 2 million
If one look at the changes within last few years in unemployment rate, the number of jobs, people who joined/left the labor market etc. there's no way that the number of people, who left Poland and haven't came back yet is bigger than one million and obviously not all of them went to the UK.
The huge numbers appear because to make It look more sensational trash media ignore a fact that these people don't stay there forever and at least half of people, who went there in 2004 are gone long time ago, they also count the way - there's X of them, let's add their children..., when in fact hardly any Poles bring children to the UK, once I even saw to the number of "Polish immigrants" added British citizens with one Polish grandparent, which obviously is a total nonsense.