A transferrable utility game is one where there’s a single resource (like dollars) where everyone’s utility is linear in that resource
For humans, money does not seem to have linear returns of utility. For what real agents could it?
My expectation for the U of an aligned AGI would be something like, the sum of the desires of humans, which, if the constituent terms have diminishing returns on resources, will also be diminishing. I can see arguments that many probable unaligned AGI might get linear returns on resources… but if humanity is involved in the negotiation (and you really hope we are) then doesn’t that still break shapley? I guess you could still potentially use shapley for analyzing the valence of ecosystems of unaligned AGI, which would be useful for comparing risk of unaligned singletons to unaligned multipolar outcomes and to authoritarian lockin, but it’s not exciting, and… actually, everything collapses to aligned-somehwhataligned-unaligned multipolar under the Grabby Aliens model.
There does seem to be a value within the human utility function that does scale linearly with resources (a variable that population ethicists and early longtermists love), but it’s not clear at all what its relationship with other variables is. Another way of phrasing this objection is, there is a lot of personal low-hanging fruit that a human has to grab before the scalable variable will be all they have left to optimize, and I don’t think I’ve ever heard of a person who gathered “enough” of the low-hanging fruit of the good life, that they started acting in a purely longtermist or stewardly way. EG, Elon will still want his space adventure no matter how many of his friends explain why they think it’s not cost-effective for reducing existential risk. Many people tell this story where, once a person is rich enough and “has their needs met”, they’re supposed to optimize the scalable term and consequently become selfless (the self is finite, even in the extremes, due to the light speed limit, so anything that keeps scaling has to be a kind of selflessness), many people would like that to be true. It’s not obviously actually true, it doesn’t accord with present human behavior, and I’m not sure how to investigate it.
For humans, money does not seem to have linear returns of utility. For what real agents could it?
My expectation for the U of an aligned AGI would be something like, the sum of the desires of humans, which, if the constituent terms have diminishing returns on resources, will also be diminishing. I can see arguments that many probable unaligned AGI might get linear returns on resources… but if humanity is involved in the negotiation (and you really hope we are) then doesn’t that still break shapley? I guess you could still potentially use shapley for analyzing the valence of ecosystems of unaligned AGI, which would be useful for comparing risk of unaligned singletons to unaligned multipolar outcomes and to authoritarian lockin, but it’s not exciting, and… actually, everything collapses to aligned-somehwhataligned-unaligned multipolar under the Grabby Aliens model.
There does seem to be a value within the human utility function that does scale linearly with resources (a variable that population ethicists and early longtermists love), but it’s not clear at all what its relationship with other variables is. Another way of phrasing this objection is, there is a lot of personal low-hanging fruit that a human has to grab before the scalable variable will be all they have left to optimize, and I don’t think I’ve ever heard of a person who gathered “enough” of the low-hanging fruit of the good life, that they started acting in a purely longtermist or stewardly way. EG, Elon will still want his space adventure no matter how many of his friends explain why they think it’s not cost-effective for reducing existential risk. Many people tell this story where, once a person is rich enough and “has their needs met”, they’re supposed to optimize the scalable term and consequently become selfless (the self is finite, even in the extremes, due to the light speed limit, so anything that keeps scaling has to be a kind of selflessness), many people would like that to be true. It’s not obviously actually true, it doesn’t accord with present human behavior, and I’m not sure how to investigate it.