If someone has stated they won’t do it if someone asks them not to, and his goal was for there to be fewer such comments in the future (which is the general goal of downvoting things), then it would be more efficient in terms of achieving that goal to lead by simply asking politely not to, and maybe add the “and I’ll downvote anyone who does” as a postscript.
Leading with the downvote threat seemed needlessly belligerent.
The “passive-aggressive” bit, in my opinion, was where he solicited people’s opinion on whether it offends them, and you skipped past actually saying it offends you and went to threats.