Omega appears and asks two human players (who are at least as skilled as Eliezer and Nesov) to each design an AI. The AIs will each undergo some single-player challenges like Newcomb’s Problem and Counterfactual Mugging, but there will be a one-shot PD between the two AIs at the end, with their source codes hidden from each other.
It might be helpful to consider a simpler, less ambiguous, version of this problem. Suppose the original players aren’t humans but AIs. What are the outcomes of this game given the following players:
2 AIs running CDT
2 AIs running TDT
1 AI running CDT and 1 AI running TDT
2 AIs selected randomly from {CDT, TDT} according to some distribution
assuming the types of players or the distribution are common knowledge. It seems important to give a full formal solution to this problem, then perhaps we can build more intuition on top of that.
It might be helpful to consider a simpler, less ambiguous, version of this problem. Suppose the original players aren’t humans but AIs. What are the outcomes of this game given the following players:
2 AIs running CDT
2 AIs running TDT
1 AI running CDT and 1 AI running TDT
2 AIs selected randomly from {CDT, TDT} according to some distribution
assuming the types of players or the distribution are common knowledge. It seems important to give a full formal solution to this problem, then perhaps we can build more intuition on top of that.