The players should be modeled as algorithms that respond not to each other’s moves, but as algorithms responding to information about the other player’s algorithm (by constructing a strategy for responding to the other player’s moves). In particular, saying that a certain player is a “rational agent” with certain payoff in the game might sometimes fix what that player’s algorithm (of responding to info about the other player’s algorithm) is. “It’s a rational agent”, if given as info about the player’s algorithm to the other player, in the right games, may be taken as equivalent to the complete specification of this player’s source code (modulo logical uncertainty; it’s actually easier if we say “it’s a rational agent” than if we give the source code, since in the latter case we may need to also make the inference—hence you don’t even need to be Omega to read “it’s a rational agent” as exact specification of the algorithm, if the game is right). For this reason, it’s unnecessary to actually communicate the source code, if it’s a given that the players are “rational agents” and this knowledge can, by some magic, be communicated.
The game of considering counterfactuals has the goal of revealing semantics of the other player’s algorithm: player A doesn’t care what the source code of player B looks like, it only cares of what that source code does, that is how it reacts to possible players A (to what possible players A do, not to what possible players’ source code is—nobody is caring about the coding; and player A doesn’t even know what its own code does—this is what it’s currently deciding; the coding only reflects logical uncertainty).
Note that each player has to maximize the payoff across all possible opponents, and not just the “rational agents” (particularly given that the coding for the “rational agent” may be unknown, even if the game has the “rational agent”). The other player may well be a random knot of wires (that still somehow processes the info about the other player’s algorithm, maybe punishes the player if it cooperates with “rational agents”; no relation to the payoff matrix is possible at this point). The conserved resource here is expressive power: each player can’t have an arbitrary table of answers to each of the possible other player’s algorithms (the number of possible reactions to elements of set S is bigger than set S), so it’s not possible in general to “step back” and improve on one’s algorithm pointwise, for each of the possible other players (even if it were possible computationally). This suggests that many games won’t have the unique “rational player” optimum, and it won’t be enough to communicate “it’s a rational player” to fix the algorithm or facilitate effective cooperation (for example, the ultimatum game, obviously).
Compare the notion of “rational agent” (as applied to more general games than PD) with stating that it’s known about the other player that it’s in your interest to make move X in response to that player.
(A few notes, with no particular point.)
The players should be modeled as algorithms that respond not to each other’s moves, but as algorithms responding to information about the other player’s algorithm (by constructing a strategy for responding to the other player’s moves). In particular, saying that a certain player is a “rational agent” with certain payoff in the game might sometimes fix what that player’s algorithm (of responding to info about the other player’s algorithm) is. “It’s a rational agent”, if given as info about the player’s algorithm to the other player, in the right games, may be taken as equivalent to the complete specification of this player’s source code (modulo logical uncertainty; it’s actually easier if we say “it’s a rational agent” than if we give the source code, since in the latter case we may need to also make the inference—hence you don’t even need to be Omega to read “it’s a rational agent” as exact specification of the algorithm, if the game is right). For this reason, it’s unnecessary to actually communicate the source code, if it’s a given that the players are “rational agents” and this knowledge can, by some magic, be communicated.
The game of considering counterfactuals has the goal of revealing semantics of the other player’s algorithm: player A doesn’t care what the source code of player B looks like, it only cares of what that source code does, that is how it reacts to possible players A (to what possible players A do, not to what possible players’ source code is—nobody is caring about the coding; and player A doesn’t even know what its own code does—this is what it’s currently deciding; the coding only reflects logical uncertainty).
Note that each player has to maximize the payoff across all possible opponents, and not just the “rational agents” (particularly given that the coding for the “rational agent” may be unknown, even if the game has the “rational agent”). The other player may well be a random knot of wires (that still somehow processes the info about the other player’s algorithm, maybe punishes the player if it cooperates with “rational agents”; no relation to the payoff matrix is possible at this point). The conserved resource here is expressive power: each player can’t have an arbitrary table of answers to each of the possible other player’s algorithms (the number of possible reactions to elements of set S is bigger than set S), so it’s not possible in general to “step back” and improve on one’s algorithm pointwise, for each of the possible other players (even if it were possible computationally). This suggests that many games won’t have the unique “rational player” optimum, and it won’t be enough to communicate “it’s a rational player” to fix the algorithm or facilitate effective cooperation (for example, the ultimatum game, obviously).
Compare the notion of “rational agent” (as applied to more general games than PD) with stating that it’s known about the other player that it’s in your interest to make move X in response to that player.