I realized there is a problem when you have a 3-action problem, where the first 2-action agent chooses between 0 utility and passing control to the second 2-action agent, and the second 2-action agent chooses between ^{1}⁄_{2} and 1 expected utility.

The problem is that there’s a stable equilibrium where the first agent passes off control and the second agent always chooses ^{1}⁄_{2} expected utility. The second agent makes this choice because they think that, if they choose 1, then the first agent will choose 0. The probability that the first agent chooses 0 is 0 but, in some sequence of CODs leading up to the actual COD, the first agent is more likely to choose 0 if the second agent chooses 1.

Basically, irrational threats can be relevant to COEDT even though they happen with probability 0.

We could fix this with a direct construction of queries that can return more than 2 results (as in reflective oracle distributions), but in any case this is a serious problem for sequential decision problems and multi-player common-payoff games.

I realized there is a problem when you have a 3-action problem, where the first 2-action agent chooses between 0 utility and passing control to the second 2-action agent, and the second 2-action agent chooses between

^{1}⁄_{2}and 1 expected utility.The problem is that there’s a stable equilibrium where the first agent passes off control and the second agent always chooses

^{1}⁄_{2}expected utility. The second agent makes this choice because they think that, if they choose 1, then the first agent will choose 0. The probability that the first agent chooses 0 is 0 but, in some sequence of CODs leading up to the actual COD, the first agent is more likely to choose 0 if the second agent chooses 1.Basically, irrational threats can be relevant to COEDT even though they happen with probability 0.

We could fix this with a direct construction of queries that can return more than 2 results (as in reflective oracle distributions), but in any case this is a serious problem for sequential decision problems and multi-player common-payoff games.