Imagine if humanity survives for the next billion years, expands to populate the entire galaxy, has a magnificent (peaceful, complex) civilization, and is almost uniformly miserable because it consists of multiple fundamentally incompatible subgroups. Nearly everyone is essentially undergoing constant torture, because of a strange, unfixable psychological quirk that creates a powerful aversion to certain other types of people (who are all around them).
If the only alternative to that dystopian future (besides human extinction) is to exterminate some subgroup of humanity, then that creates a dilemma: torture vs. genocide. My inclination is that near-universal misery is worse than extinction, and extinction is worse than genocide.
And that seems to be where this hypothetical is headed, if you keep applying “least convenient possible world” and ruling out all of the preferable potential alternatives (like separating the groups, or manipulating either group’s genes/brains/noses to stop the aversive feelings). If you keep tailoring a hypothetical so that the only options are mass suffering, genocide, and human extinction, then the conclusion is bound to be pretty repugnant. None of those bullets are particularly appetizing but you’ll have to chew on one of them. Which bullet to bite depends on the specifics; as the degree of misery among the aversion-sufferers gets reduced from torture-levels towards insignificance at some point my preference ordering will flip.
I noticed something similar in another comment. CEV must compare the opportunity cost of pursuing a particular terminal value at the expense of all other terminal values, at least in a universe with constrained resources. This leads me to believe that CEV will suggest that the most costly (in terms of utility opportunity lost by choosing to spend time fulfilling a particular terminal value instead of another) terminal value be abandoned until only one is left and we become X maximizers. This might be just fine if X is still humane, but it seems like any X will be expressible as a conjunction of disjunctions and any particular disjuctive clause will have the highest opportunity cost and could be removed to increase overall utility, again leading to maximizing the smallest expressible (or easiest to fulfill) goal.
Classical failed scenarios. Great morphological/structural changes need legal constraints to don’t become very common, or be risk-averse, to prevent the creation of inumerous subgroups with alien values. But contra this, subgroups could go astray far enough to don’t be caugth, and make whatever change they want, even creating new subgroups to torture or kill. In this case specifically, I assume we have to deal with this problem before structural changes become common.
Imagine if humanity survives for the next billion years, expands to populate the entire galaxy, has a magnificent (peaceful, complex) civilization, and is almost uniformly miserable because it consists of multiple fundamentally incompatible subgroups. Nearly everyone is essentially undergoing constant torture, because of a strange, unfixable psychological quirk that creates a powerful aversion to certain other types of people (who are all around them).
If the only alternative to that dystopian future (besides human extinction) is to exterminate some subgroup of humanity, then that creates a dilemma: torture vs. genocide. My inclination is that near-universal misery is worse than extinction, and extinction is worse than genocide.
And that seems to be where this hypothetical is headed, if you keep applying “least convenient possible world” and ruling out all of the preferable potential alternatives (like separating the groups, or manipulating either group’s genes/brains/noses to stop the aversive feelings). If you keep tailoring a hypothetical so that the only options are mass suffering, genocide, and human extinction, then the conclusion is bound to be pretty repugnant. None of those bullets are particularly appetizing but you’ll have to chew on one of them. Which bullet to bite depends on the specifics; as the degree of misery among the aversion-sufferers gets reduced from torture-levels towards insignificance at some point my preference ordering will flip.
I noticed something similar in another comment. CEV must compare the opportunity cost of pursuing a particular terminal value at the expense of all other terminal values, at least in a universe with constrained resources. This leads me to believe that CEV will suggest that the most costly (in terms of utility opportunity lost by choosing to spend time fulfilling a particular terminal value instead of another) terminal value be abandoned until only one is left and we become X maximizers. This might be just fine if X is still humane, but it seems like any X will be expressible as a conjunction of disjunctions and any particular disjuctive clause will have the highest opportunity cost and could be removed to increase overall utility, again leading to maximizing the smallest expressible (or easiest to fulfill) goal.
Classical failed scenarios. Great morphological/structural changes need legal constraints to don’t become very common, or be risk-averse, to prevent the creation of inumerous subgroups with alien values. But contra this, subgroups could go astray far enough to don’t be caugth, and make whatever change they want, even creating new subgroups to torture or kill. In this case specifically, I assume we have to deal with this problem before structural changes become common.