The incentive is weaker than you seem to suggest. Surely, I gain nothing tangible by inducing people to tiptoe carefully around my minefield.
Yes, you do. If everything unpleasant to you causes you a huge amount of suffering instead of, say, mild annoyance, other people (utilitarians) will abstain from doing things that are unpleasant to you as the negative utility to you outweighs the positive utility to them.
Making the (flawed) assumption that in a disagreement, they cannot both be wrong.