Games for factoring out variables

All the methods proposed for factoring out (for having the AI maximise a certain value while ‘ignoring’ it’s impact via ) can be put on the same general footing. For some set , define a function on with and .

Then for a utility , the general requirements is for the AI to maximise the quantity:

  • ,

subject to some constraints on .


Let’s play a game

Define the two-player game by allowing each player to have moves in . The expected utility, for player , of the moves is defined to be . To completely define the game, we need the expected utility for the second player. We’ll just set that to ensure the game is symmetric the expected utility of player two for is the same as the expected utility for player one for .

Then is effectively a probability distribution over the choice of possible moves for . Standard symmetric games include the stag hunt, the prisoner’s dilemma, and the coordination game. For our purposes, since the actions for have meaningful labels, we’ll be considering skew coordination game given as follows:

+-----+-----+-----+ | | l | r | +-----+-----+-----+ | l |(0,0)|(1,1)| +-----+-----+-----+ | r |(1,1)|(0,0)| +-----+-----+-----+

These ‘games’ will be mainly used to show that the various methods reach different solutions in different situations, hence that they are genuinely different methods.

Standard maximalisation

If we constrain if , then that equation becomes:

This makes into the action distribution over the AI must choose from to maximise in the classical sense. In terms of , this means the AI will make the superrational choice between the two players, if they lack any way of distinguishing themselves. Thus it will pick for the stag hunt, in the prisoner’s dilemma, and or (or some mixture between the two) in the skew coordination game.

Double variable maximalisation

First consider that there are no additional constraints on . The AI can pick the values to be whatever it wants. It therefore acts like a dictator in , choosing both player’s outcomes. Under this, it will pick for the stag hunt, in the prisoner’s dilemma, or (or some mixture between the two) in the skew coordination game. Note that the AI can always achieve maximal utility by choosing a single action for both players. Thus if we added the restriction that , it could still reach at least one of these optimums.

What if we added the restriction for a single ? Maximising this over possible is equivalent with the AI reaching the superrational decision for both players, where they have access to individual but not joint sources of randomness. Thus it will pick for the stag hunt, in the prisoner’s dilemma, and in the skew coordination game.

For the next case, consider that , as above, but the AI is not aiming to maximise , but to pick an that ensures

  • for all .

This is Chris and Jessica’s method. You can transform that requirement into a quantity to be maximised, but it’s not particularly illuminating. What’s more illuminating is that the that solve this are precisely the symmetric Nash equilibriums to (all symmetric games have at least one symmetric Nash equilibrium). ensures they must be symmetric, the condition on picking ensures the first player cannot improve their expectation unilaterally, and, since itself is symmetric, the second player cannot improve their expectation unilaterally.

In cases where there are multiple symmetric Nash equilibriums, we may as well have the AI choose the one that maximises the . Thus the AI will choose for the stag hunt, in the prisoner’s dilemma, and in the skew coordination game.

Single variable maximalisation

Here where is some fixed distribution. There are some obvious candidates for that—maybe one action is a default action, or is uniform across all actions. There are some more complicated methods to assign probabilities in a way that is sensible if there are multiple branching decisions.

In this case, the AI will pick or for the stag hunt, depending on and the exact rewards of the game, will always choose in the prisoner’s dilemma, and will choose if and if in the skew coordination game.

Summary

There are thus five methods, clearly distinguished by making different choices in the different games:

  1. unless . is maximalised.

  2. There are no constraints on , or . is maximalised.

  3. and is maximalised.

  4. and for all .

  5. for fixed and is maximalised.

Arbitrarily ‘bad’ decisions

All the methods (except for the first one) can reach arbitrarily bad decision in terms of real expected utility, as compared with standard expected utility maximalisation. Consider the following extension of the skew coordination problem, for large :

+-----+---------+---------+---------+ | | l | r | c | +-----+---------+---------+---------+ | l | (-W,-W) |(W+1,W+1)| (0,0) | +-----+---------+---------+---------+ | r |(W+1,W+1)| (-W,-W) | (0,0) | +-----+---------+---------+---------+ | c | (0,0) | (0,0) | (-1,-1) | +-----+---------+---------+---------+

All alternative methods will choose actions from among and only. This condemns them to a utility of , while the best action choice, , has an expected utility of .

Further considerations

To distinguish which method we should be using, issues like stability, self-consistency, and other properties will probably be needed.