So when I hear or see it written - Oh, it's just semantics - then I am not sure whether the person has no clue what semantics actually means, or whether the scientific and artsy worlds never intersect.
For sure they're not thinking about the semantic web or anything technical like that :) I think you might be overcomplicating the issue here (unless you're programming an AI to say 'it's just semantics' at the appropriate moment :)). It's a safe bet that 99% of the users of this phrase have no intention of dragging up any deeper, specialist meanings when they use it.
Using quasi-specialist words in frankly banal contexts is simply the stock-in-trade of linguistic aspiration. There are hundreds of other similar phrases in English, but permit me as an Anglo to note the two expressions in Polish that particularly jump out at me (even now, after all this time):
profilaktycznie - 'just in case' is often all that is meant here. In English, outside of medical journals, prophylactic generally means birth control.
apodyktyczny - likewise, used to mean little more than 'irritable' or 'dismissive'. But ahhhhh, that Greek etymology,
proszę państwo!
Mafketis is on the money here with
to kwestia semantyki, I reckon - but I don't think you need to 'translate' the deeper meanings as a) the surface one is 99% of it and b) the whole point about implied meaning is it's context-sensitive and so not verbalised.
EDIT: relax about your standard of written English. I aspire to, but fall considerably short of, that level in Polish. That aside, many, and probably most, native English speakers can't communicate at that level. Round of applause from me.