Szkoda mi czasu!
not sure if I've ever heard that, what I do frequently hear is 'szkoda czasu' "(it's a) waste of time"
there's also 'kłamać w żywe oczy' "to lie (straight) to someone's face"
'can we talk one to one'
I'd usually translate "w cztery oczy" as 'face to face' or 'directly' or with your example even "Can we talk about this by ourselves?"