Obviously CEV isn’t identical to preference utilitarianism, but CEV and preference utilitarianism have the following principles in common, which the hack exploits:
Give people what they want, instead of what you think is good for them.
If different people want different things, give each individual equal weight.
It seems clear that Eliezer got these ideas from preference utilitarianism, and they share some of the same flaws as a result.
Obviously CEV isn’t identical to preference utilitarianism, but CEV and preference utilitarianism have the following principles in common, which the hack exploits:
Give people what they want, instead of what you think is good for them.
If different people want different things, give each individual equal weight.
It seems clear that Eliezer got these ideas from preference utilitarianism, and they share some of the same flaws as a result.