I don’t think a single downvote is that significant. I’ve seen so many inexplicable downvotes that I would suspect there’s a bot that every time a new comment is posted generates a random number between 0 and 1 and downvotes the comment if the random number is less than 0.05 -- if I could see any reason why anyone would do that.
Now that you’ve suggested it...