If you do happen to think that there is a source of morality beyond human beings… and I hear from quite a lot of people who are happy to rhapsodize on how Their-Favorite-Morality is built into the very fabric of the universe… then what if that morality tells you to kill people?
If you believe that there is any kind of stone tablet in the fabric of the universe, in the nature of reality, in the structure of logic—anywhere you care to put it—then what if you get a chance to read that stone tablet, and it turns out to say “Pain Is Good”? What then?
Maybe you should hope that morality isn’t written into the structure of the universe. What if the structure of the universe says to do something horrible?
And if an external objective morality does say that the universe should occupy some horrifying state… let’s not even ask what you’re going to do about that. No, instead I ask: What would you have wished for the external objective morality to be instead? What’s the best news you could have gotten, reading that stone tablet?
Go ahead. Indulge your fantasy. Would you want the stone tablet to say people should die of old age, or that people should live as long as they wanted? If you could write the stone tablet yourself, what would it say?
Maybe you should just do that?
I mean… if an external objective morality tells you to kill people, why should you even listen?
The Moral Void, Eliezer Yudkowsky
What’s more important to you, your desire to prevent genocide or your desire for a simple consistent utility function?