A theorem that, as far as I know, isn’t in the collective LW memescape and really should be, is Harsanyi’s theorem. Loosely, it states that any decision-making procedure for making decisions on behalf of a collection of VNM agents satisfying a few straightforward axioms must be equivalent to maximizing a weighted sum of the VNM agents’ utility functions. The sum has to be weighted since VNM utility functions are only well-defined up to a positive affine transformation, and Harsanyi’s theorem doesn’t provide any guidance in determining the weights. I don’t know that I’ve seen a public discussion anywhere of the ramifications of this.
Harsanyi’s theorem only applies to VNM agents whose beliefs about the world agree. In the general case where they don’t, Critch proved a generalization in this paper which I also haven’t seen a public discussion of the ramifications of. I haven’t digested the details, but the gist is that how much weight your utility function gets increases the better your beliefs predict reality.
A while ago I described something similar on the decision-theory-workshop mailing list, with two differences (I think they are improvements but YMMV):
1) It’s easier to use UDT than VNM, because UDT agents have no beliefs, only a utility function that mixes beliefs with values (e.g. if you think a coin is/was fair, your utility is the average of your utilities in heads-world and tails-world). When two agents merge, you just take a weighted sum of their utility functions.
2) In some games, specifying the weighted sum is not enough. For example, in the dividing the dollar game, maximizing any weighted sum will give the whole dollar to one player, except the equally weighted sum which is indifferent. To achieve e.g. an equal division of the dollar, the agents need to jointly observe a coinflip while constructing the merged agent, or more generally to agree on a probability distribution over merged agents (with the restriction that it must lie on the Pareto frontier).
I have had some conversations about this in Berkeley and at FHI, and I think I remember some posts by Stuart Armstrong on this. So this hasn’t fully avoided the landscape, though I agree that I haven’t seen any particularly good coverage of this.
A theorem that, as far as I know, isn’t in the collective LW memescape and really should be, is Harsanyi’s theorem. Loosely, it states that any decision-making procedure for making decisions on behalf of a collection of VNM agents satisfying a few straightforward axioms must be equivalent to maximizing a weighted sum of the VNM agents’ utility functions. The sum has to be weighted since VNM utility functions are only well-defined up to a positive affine transformation, and Harsanyi’s theorem doesn’t provide any guidance in determining the weights. I don’t know that I’ve seen a public discussion anywhere of the ramifications of this.
Harsanyi’s theorem only applies to VNM agents whose beliefs about the world agree. In the general case where they don’t, Critch proved a generalization in this paper which I also haven’t seen a public discussion of the ramifications of. I haven’t digested the details, but the gist is that how much weight your utility function gets increases the better your beliefs predict reality.
A while ago I described something similar on the decision-theory-workshop mailing list, with two differences (I think they are improvements but YMMV):
1) It’s easier to use UDT than VNM, because UDT agents have no beliefs, only a utility function that mixes beliefs with values (e.g. if you think a coin is/was fair, your utility is the average of your utilities in heads-world and tails-world). When two agents merge, you just take a weighted sum of their utility functions.
2) In some games, specifying the weighted sum is not enough. For example, in the dividing the dollar game, maximizing any weighted sum will give the whole dollar to one player, except the equally weighted sum which is indifferent. To achieve e.g. an equal division of the dollar, the agents need to jointly observe a coinflip while constructing the merged agent, or more generally to agree on a probability distribution over merged agents (with the restriction that it must lie on the Pareto frontier).
I have had some conversations about this in Berkeley and at FHI, and I think I remember some posts by Stuart Armstrong on this. So this hasn’t fully avoided the landscape, though I agree that I haven’t seen any particularly good coverage of this.