On the contrary, I say the FAI shouldn’t help because Steve wants to live more than Fred wants to kill him.
Doesn’t this fall victim to utility monsters, though? If there’s some actor who wants to kill you more than you want not to die, then the FAI would be obliged to kill you. That’s a classic utility monster: an entity that wants harder than anyone else.
One solution is to renorm everyone’s wants, such that the desires of any sentient being don’t count any more than any other. But this leads directly to Parfit’s Repugnant Conclusion¹, or at least to some approximation thereof: maximizing the number of sentient beings, even at the expense of their individual wants or pleasure.
¹ Parfit’s repugnant conclusion, Level 7 Necromancer spell. Material components include the severed head of a utility monster, or a bag of holding filled with orgasmium, which is consumed during casting. Fills a volume of space with maximally dense, minimally happy (but barely nonsuicidal) sentient minds, encoded onto any available material substrate.
I agree that you have to renorm everyone’s wants for this to work. I also agree that if you can construct broken minds for the purpose of manipulating the FAI, we need provisions to guard against that. My preferred alternative at the moment follows:
Before people become able to construct broken minds, the FAI cares about everything that’s genetically human.
After we find the first genetically human mind deliberately broken for the purpose of manipulating the FAI, we guess when the FAI started to be influenced by that, and retroactive to just before that time we introduce a new policy: new individuals start out with a weight of 0, and can receive weight transferred from their parents, so the total weight is conserved. I don’t want an economy of weight-transfer to arise, so it would be a one-way irreversible transfer.
This might lead to a few people running around with a weight of 0 because their parents never made the transfer. This would be suboptimal, but it would not have horrible conclusions because the AI would care for the parents who probably care for the new child, so the AI would in effect care some for the new child.
Death of the parents doesn’t break this. Caring about the preference of dead people is not a special case.
I encourage people to reply to this post with bugs in this alternative or with other plausible alternatives.
Doesn’t this fall victim to utility monsters, though? If there’s some actor who wants to kill you more than you want not to die, then the FAI would be obliged to kill you. That’s a classic utility monster: an entity that wants harder than anyone else.
One solution is to renorm everyone’s wants, such that the desires of any sentient being don’t count any more than any other. But this leads directly to Parfit’s Repugnant Conclusion¹, or at least to some approximation thereof: maximizing the number of sentient beings, even at the expense of their individual wants or pleasure.
¹ Parfit’s repugnant conclusion, Level 7 Necromancer spell. Material components include the severed head of a utility monster, or a bag of holding filled with orgasmium, which is consumed during casting. Fills a volume of space with maximally dense, minimally happy (but barely nonsuicidal) sentient minds, encoded onto any available material substrate.
I agree that you have to renorm everyone’s wants for this to work. I also agree that if you can construct broken minds for the purpose of manipulating the FAI, we need provisions to guard against that. My preferred alternative at the moment follows:
Before people become able to construct broken minds, the FAI cares about everything that’s genetically human.
After we find the first genetically human mind deliberately broken for the purpose of manipulating the FAI, we guess when the FAI started to be influenced by that, and retroactive to just before that time we introduce a new policy: new individuals start out with a weight of 0, and can receive weight transferred from their parents, so the total weight is conserved. I don’t want an economy of weight-transfer to arise, so it would be a one-way irreversible transfer.
This might lead to a few people running around with a weight of 0 because their parents never made the transfer. This would be suboptimal, but it would not have horrible conclusions because the AI would care for the parents who probably care for the new child, so the AI would in effect care some for the new child.
Death of the parents doesn’t break this. Caring about the preference of dead people is not a special case.
I encourage people to reply to this post with bugs in this alternative or with other plausible alternatives.