Thank you, everyone, for the input!
These people are messing you around and what they're doing isn't normal. In your rental agreement there should be a notice period (
okres wypowiedzenia), usually a month, maybe 2 in rare cases. You should give notice, explaining that you want a flat where they don't mess around with the water supply or expect you to tiptoe around.
My landlord (who owns all the units in the building) told me that is the norm because leaving it on 24/7 would spoil the water pump.
He is lying to you. I've rented 7 flats in Poland, arranged rentals for other people and rent one out now and have never ever heard of anything like this.