why should one group of people have the right to dictate to other persons which legal activities they can allow on their property?
Why is something legal or illegal in the first place? Because a group of people passed legislation outlining the rights of other people/groups/organisations. So by the very virtue of living in a governed non-anarchistic state, you are living in a society that determines that groups of people dictate the standards by which others live.
You are right smoking in public/private places in many countries is currently legal, but it is being faded out. When you consider that smoking would be in the top 3 causes for health expenditure in most western countries, I would expect that the average tax payer has a reason to want to see change.
I would think that once upon a time there were schools/hospitals that you could smoke and now you no longer can? A law was passed that the health of the person not smoking was important enough to outlaw others having the "right" to impact them.
Governments are essentially law makers, so of course they have a right to legalize or criminalize an activity as they see fit, if it is unjust then people will rise up and vote them out, I doubt this has happened often over smoking laws, if at all.
Take the following as an example..
Once opium houses were legal in Australia. It was no issue to go to one of these houses and get as high as a kite. Now obviously over time as the government (and most likely pushed by public demand) has decided that these establishments where unhealthy/dangerous etc, they have outlawed them and the practice in general. I would suggest it would not be at all stretching the truth to suggest that people would have been indignant and wondered themselves what gave the government the right to ban this "legal" activity.
Whilst everyone has the right to challenge through their local government laws to be enacted and in fact can of course run and attempt to overturn legislation themselves... questioning the right of a group of people to change or implement laws seems a strange thing to be questioning when it is the fundamental basis of the society most of us live in.