I’m inclined to agree, but then “I’m certain Bob’s utility function is not literally that” is an assumption about alignment between people. Maybe it’s a justified assumption (what some would call an “axiom”), but it is an assumption.
Moreover, though, I think even slightly stronger forms of this assumption are false. Like, it is not rare for people to think that, for certain values of person X, “A thing that is strictly positive for person X—that makes them happier, healthier, or more capable—is a bad thing.” Values of person X for which I think there are at least some people who endorse that statement include: “the dictator of an oppressive country who keeps ordering his secret police to kill his political opponents”, “a general for an army that’s aggressively invading my country”, “a convicted murderer”… moving in the more controversial direction: “a person I’m sure is a murderer but isn’t convicted”, “a regular citizen of an oppressive country, whose economic output (forcibly) mostly ends up in the hands of the evil dictator” (I think certain embargoes have had rationales like this).
I think there are people who believe our society sucks, that the only and inevitable path to improvement is a Revolution that will bring us to a glorious utopia, and that people being unhappy will make the Revolution sooner (and thus be net beneficial), and therefore they consider it a negative for anything to make anyone in the entire society happier. (And, more mundanely, I think there are people who see others enjoying themselves and feel annoyed by it, possibly because it reminds them of their own pain or something.)
Each of those can be debated on its own merits (e.g. one could claim that, if the dictator becomes happier, he might ease off, or that if his health declines he might get desperate or insane and do something worse; and obviously the “accelerate the revolution” strategy is extremely dangerous and I’m not convinced it’s ever the right idea), but the point is, there are people with those beliefs and those preferences.
You can do something like declaring a bunch of those preferences out of bounds—that our society will treat them like they don’t exist. (The justice system could be construed as saying “We’ll only endorse a preference for ‘negative utility for person X’ when person X is duly convicted of a crime, with bounds on exactly how negative we go.”) I think this is a good idea, and that this lets you get some good stuff done. But it is a step you should be aware that you’re taking.
I’m inclined to agree, but then “I’m certain Bob’s utility function is not literally that” is an assumption about alignment between people. Maybe it’s a justified assumption (what some would call an “axiom”), but it is an assumption.
Moreover, though, I think even slightly stronger forms of this assumption are false. Like, it is not rare for people to think that, for certain values of person X, “A thing that is strictly positive for person X—that makes them happier, healthier, or more capable—is a bad thing.” Values of person X for which I think there are at least some people who endorse that statement include: “the dictator of an oppressive country who keeps ordering his secret police to kill his political opponents”, “a general for an army that’s aggressively invading my country”, “a convicted murderer”… moving in the more controversial direction: “a person I’m sure is a murderer but isn’t convicted”, “a regular citizen of an oppressive country, whose economic output (forcibly) mostly ends up in the hands of the evil dictator” (I think certain embargoes have had rationales like this).
I think there are people who believe our society sucks, that the only and inevitable path to improvement is a Revolution that will bring us to a glorious utopia, and that people being unhappy will make the Revolution sooner (and thus be net beneficial), and therefore they consider it a negative for anything to make anyone in the entire society happier. (And, more mundanely, I think there are people who see others enjoying themselves and feel annoyed by it, possibly because it reminds them of their own pain or something.)
Each of those can be debated on its own merits (e.g. one could claim that, if the dictator becomes happier, he might ease off, or that if his health declines he might get desperate or insane and do something worse; and obviously the “accelerate the revolution” strategy is extremely dangerous and I’m not convinced it’s ever the right idea), but the point is, there are people with those beliefs and those preferences.
You can do something like declaring a bunch of those preferences out of bounds—that our society will treat them like they don’t exist. (The justice system could be construed as saying “We’ll only endorse a preference for ‘negative utility for person X’ when person X is duly convicted of a crime, with bounds on exactly how negative we go.”) I think this is a good idea, and that this lets you get some good stuff done. But it is a step you should be aware that you’re taking.