Why do we need a single CEV value system? A FAI can calculate as many value systems as it needs and keep incompatible humans separate. Group size is just another parameter to optimize. Religious fundamentalists can live in their own simulated universe, liberals in another.
Upvoting back to zero because I think this is an important question to address.
If I prefer that people not be tortured, and that’s more important to me than anything else, then I ought not prefer a system that puts all the torturers in their own part of the world where I don’t have to interact with them over a system that prevents them from torturing.
More generally, this strategy only works if there’s nothing I prefer/antiprefer exist, but merely things that I prefer/antiprefer to be aware of.
[T]here’s nothing I prefer/antiprefer exist, but merely things that I prefer/antiprefer to be aware of.
is a conceivable extrapolation from a starting point where you antiprefer something’s existence (in the extreme, with MWI you may not have much say what does/doesn’t exist, just how much of it in which branches).
It’s also possible that you hold both preferences (prefer X not exist, prefer not to be aware of X) and the existence preference gets dropped for being incompatible with other values held by other people while the awareness preference does not.
Assuming 100% isolation it would be indistinguishable from living in a universe where the Many Worlds Interpretation is true, but it still seems wrong. The FAI could consider avoiding groups whose even theoretical existence could cause offence, but I don’t see any good way to assign weight to this optimization pressure.
Even so, I think splitting humanity into multiple groups is likely to be a better outcome than a single group. I don’t consider the “failed utopia” described in http://lesswrong.com/lw/xu/failed_utopia_42/ to be particularly bad.
The failed utopia is better than our current world, certainly. But the genie isn’t Friendly.
In principle, I could interact with the immoral cluster. AI’s interference is not relevant to the morality of the situation because I was part of the creation of the AI. Otherwise, I would be morally justified in ignoring the suffering in some distant part of the world because it will have no practical impact on my life. By contrast, I simply cannot interact with other branches under the MWI—it’s a baked in property of the universe that I never had any input into.
Why do we need a single CEV value system? A FAI can calculate as many value systems as it needs and keep incompatible humans separate. Group size is just another parameter to optimize. Religious fundamentalists can live in their own simulated universe, liberals in another.
Upvoting back to zero because I think this is an important question to address.
If I prefer that people not be tortured, and that’s more important to me than anything else, then I ought not prefer a system that puts all the torturers in their own part of the world where I don’t have to interact with them over a system that prevents them from torturing.
More generally, this strategy only works if there’s nothing I prefer/antiprefer exist, but merely things that I prefer/antiprefer to be aware of.
It’s a potential outcome, I suppose, in that
is a conceivable extrapolation from a starting point where you antiprefer something’s existence (in the extreme, with MWI you may not have much say what does/doesn’t exist, just how much of it in which branches).
It’s also possible that you hold both preferences (prefer X not exist, prefer not to be aware of X) and the existence preference gets dropped for being incompatible with other values held by other people while the awareness preference does not.
The child molester cluster (where they grow child simply to molest them, then kill them) doesn’t bother you, even if you never interact with it?
Because I’m fairly certain I wouldn’t like what CEV(child molester) would output and wouldn’t want an AI to implement it.
Assuming 100% isolation it would be indistinguishable from living in a universe where the Many Worlds Interpretation is true, but it still seems wrong. The FAI could consider avoiding groups whose even theoretical existence could cause offence, but I don’t see any good way to assign weight to this optimization pressure.
Even so, I think splitting humanity into multiple groups is likely to be a better outcome than a single group. I don’t consider the “failed utopia” described in http://lesswrong.com/lw/xu/failed_utopia_42/ to be particularly bad.
Well, not if “child-molesters” and “non-child-molestors” are competing for limited resources.
The failed utopia is better than our current world, certainly. But the genie isn’t Friendly.
In principle, I could interact with the immoral cluster. AI’s interference is not relevant to the morality of the situation because I was part of the creation of the AI. Otherwise, I would be morally justified in ignoring the suffering in some distant part of the world because it will have no practical impact on my life. By contrast, I simply cannot interact with other branches under the MWI—it’s a baked in property of the universe that I never had any input into.
What if space travel turns out to be impossible, and the superintelligence has to allocate the limited computational resources of the solar system?