Cooperate (unless paperclip decides that Earth is dominated by traditional game theorists...)
The standard argument looks like this (let’s forget about the Nash equilibrium endpoint for a moment):
(1) Arbiter: let’s (C,C)!
(2) Player1: I’d rather (D,C).
(3) Player2: I’d rather (D,D).
(4) Arbiter: sold!
The error is that this incremental process reacts on different hypothetical outcomes, not on actual outcomes. This line of reasoning leads to the outcome (D,D), and yet it progresses as if (C,C) and (D,C) were real options of the final outcome. It’s similar to the Unexpected hanging paradox: you can only give one answer, not build a long line of reasoning where each step assumes a different answer.
It’s preferrable to choose (C,C) and similar non-Nash equilibrium options in other one-off games if we assume that other player also bets on cooperation. And he will do that only if he assumes that first player does the same, and so on. This is a situation of common knowledge. How can Player1 come to the same conclusion as Player2? They search for the best joint policy that is stable under common knowledge.
Let’s extract the decision procedures selected by both sides to handle this problem as self-contained policies, P1 and P2. Each of these policies may decide differently depending on what policy another player is assumed to use. The stable set of policies is where there is no thrashing, when P1=P1(P2) and P2=P2(P1). Players don’t select outcomes, but policies, where policy may not reflect player’s preferences, but joint policy (P1,P2) that players select is a stable policy that is preferable to other stable policies for each player. In our case, both policies for (C,C) are something like “decide self.C; if other.D, decide self.D”. Works like iterated prisoner’s dilemma, but without actual iteration, iteration happens in the model when it needs to be mutually accepted.
(I know it’s somewhat inconclusive, couldn’t find time to pinpoint it better given a time limit, but I hope one can construct a better argument from the corpse of this one.)
Cooperate (unless paperclip decides that Earth is dominated by traditional game theorists...)
The standard argument looks like this (let’s forget about the Nash equilibrium endpoint for a moment): (1) Arbiter: let’s (C,C)! (2) Player1: I’d rather (D,C). (3) Player2: I’d rather (D,D). (4) Arbiter: sold!
The error is that this incremental process reacts on different hypothetical outcomes, not on actual outcomes. This line of reasoning leads to the outcome (D,D), and yet it progresses as if (C,C) and (D,C) were real options of the final outcome. It’s similar to the Unexpected hanging paradox: you can only give one answer, not build a long line of reasoning where each step assumes a different answer.
It’s preferrable to choose (C,C) and similar non-Nash equilibrium options in other one-off games if we assume that other player also bets on cooperation. And he will do that only if he assumes that first player does the same, and so on. This is a situation of common knowledge. How can Player1 come to the same conclusion as Player2? They search for the best joint policy that is stable under common knowledge.
Let’s extract the decision procedures selected by both sides to handle this problem as self-contained policies, P1 and P2. Each of these policies may decide differently depending on what policy another player is assumed to use. The stable set of policies is where there is no thrashing, when P1=P1(P2) and P2=P2(P1). Players don’t select outcomes, but policies, where policy may not reflect player’s preferences, but joint policy (P1,P2) that players select is a stable policy that is preferable to other stable policies for each player. In our case, both policies for (C,C) are something like “decide self.C; if other.D, decide self.D”. Works like iterated prisoner’s dilemma, but without actual iteration, iteration happens in the model when it needs to be mutually accepted.
(I know it’s somewhat inconclusive, couldn’t find time to pinpoint it better given a time limit, but I hope one can construct a better argument from the corpse of this one.)