We’re trying to create an intelligent being that acts morally when it doesn’t have to. The Orthogonality Thesis says that’s possible, and intuition says it’s hard. For a selfish being, what evolution produces, that can’t be based simply on game theory: the game theory says “you can do what you want and there are no consequences”. Game theory looks like the second column in your table. What we want is a being that isn’t selfish at all, but “otherish”: whose utility function is aligned to our utility function. That’s even better than your Omega proposal: that’s a being whose payoff is −10m for pressing the button and 0m for not pressing the button: it ignores the cash entirely (or actually, would donate it to charity, in which case its payoff for pressing the button is −9m, matching your Omega). That’s a being whose utility function is that of an intelligent piece of humanity’s extended phenotype: something that could not evolve, but is the correct thing for us to build.
How does Evolutionary Moral Psychology help here? Well, I have a post about that…
We’re trying to create an intelligent being that acts morally when it doesn’t have to. The Orthogonality Thesis says that’s possible, and intuition says it’s hard. For a selfish being, what evolution produces, that can’t be based simply on game theory: the game theory says “you can do what you want and there are no consequences”. Game theory looks like the second column in your table. What we want is a being that isn’t selfish at all, but “otherish”: whose utility function is aligned to our utility function. That’s even better than your Omega proposal: that’s a being whose payoff is −10m for pressing the button and 0m for not pressing the button: it ignores the cash entirely (or actually, would donate it to charity, in which case its payoff for pressing the button is −9m, matching your Omega). That’s a being whose utility function is that of an intelligent piece of humanity’s extended phenotype: something that could not evolve, but is the correct thing for us to build.
How does Evolutionary Moral Psychology help here? Well, I have a post about that…