There’s a relevant footnote in the next post: while it’s possible to write agents that punish a particular decision theory, an agent that’s trying (in any meaningful sense) to maximize its stated payoffs won’t do this, and it’s certainly not a constant strategy (i.e. a one-player game) either. In that sense, we can say that such problems are less likely to be encountered in reality.
It might appear as if they’re playing, for example, Prisoners’ Dilemna, but with access to each other’s source code they’re really playing some sort of Prisoners’ Meta-Dilemna, a fundamentally different game for all that it might seem superficially similar.
I agree, in the same way that Iterated Prisoners’ Dilemma (with no source code access) is a fundamentally different game from One-Shot Prisoners’ Dilemma (with no source code access). But there are real-life instances that are closer to the source-code access version than to the classical game theory version- for instance, a Cold War between two powers, each of which has a network of spies in the other’s government.
There’s a relevant footnote in the next post: while it’s possible to write agents that punish a particular decision theory, an agent that’s trying (in any meaningful sense) to maximize its stated payoffs won’t do this, and it’s certainly not a constant strategy (i.e. a one-player game) either. In that sense, we can say that such problems are less likely to be encountered in reality.
I agree, in the same way that Iterated Prisoners’ Dilemma (with no source code access) is a fundamentally different game from One-Shot Prisoners’ Dilemma (with no source code access). But there are real-life instances that are closer to the source-code access version than to the classical game theory version- for instance, a Cold War between two powers, each of which has a network of spies in the other’s government.