[The following argument is made with tongue somewhat in cheek.]
Rationality (with a capital R) is supposed to be an idealized algorithm that is universally valid for all agents. Therefore, this algorithm, as such, doesn’t know whether it will be instantiated in Player One (P1) or Player Two (P2). Yes, each player knows which one they are. But this knowledge is input that they feed to their Rationality subroutine. The subroutine itself doesn’t come with this knowledge built in.
Since Rationality doesn’t know where it will end up, it doesn’t know which outcomes will maximize its utility. Thus, the most rational thing for Rationality (qua idealized algorithm) to do is to precommit to strategies for P1 and P2 that maximize its expected utility given this uncertainty.
That is, let EU₁(s, t) be the expected utility to P1 if P1 implements strategy s and P2 implements strategy t. Define EU₂(s, t) similarly. Then Rationality wants to precommit the players to the respective strategies s and t that maximize
p EU₁(s, t) + q EU₂(s, t),
where p (respectively, q) is the measure of the copies of Rationality that end up in P1 (respectively, P2).
Assuming that p = q, Rationality will therefore have P1 choose C (with probability 1) and have P2 choose Y (with probability 1).
That’s all well and good, but will the players actually act that way? After all, they are stipulated to care only about themselves, so P2 in particular will not want to act according to this selfless strategy.
Yes, but this selfish desire on P2′s part is not built into P2′s Rationality subroutine (call it R), because Rationality is universal. P2′s selfish desires must be implemented elsewhere, outside of R. To be sure, P2 is free to feed the information that it is P2 to R, but R won’t do anything with this information, because R is already precommitted to a strategy for the reasons given above.
And since the players are given to be rational, they are forced to act according to the strategies pre-selected by their Rationality subroutines, despite their wishes to the contrary. Therefore, they will in fact act as Rationality determined.
[If this comment has any point, it is that there is a strong tension, if not a contradiction, between the idea of rationality as a universally valid mode of reasoning, on the one hand, and the idea of rational agents whose revealed preferences are selfish, on the other.]
It is true that adding different utility functions is in general an error. However, for agents bound to follow Rationality (and Rationality alone), the different utility functions are best thought of as the same utility function conditioned on different hypotheses, where the different hypotheses look like “The utility to P2 turns out to be what really matters”.
After all, if the agents are making their decisions on the basis of Rationality alone, then Rationality alone must have a utility function. Since Rationality is universal, the utility function must be universal. What alternative does Rationality have, given the constraints of the problem, other than a weighted sum of the utility functions of the different individuals who might turn out to matter?
[The following argument is made with tongue somewhat in cheek.]
Rationality (with a capital R) is supposed to be an idealized algorithm that is universally valid for all agents. Therefore, this algorithm, as such, doesn’t know whether it will be instantiated in Player One (P1) or Player Two (P2). Yes, each player knows which one they are. But this knowledge is input that they feed to their Rationality subroutine. The subroutine itself doesn’t come with this knowledge built in.
Since Rationality doesn’t know where it will end up, it doesn’t know which outcomes will maximize its utility. Thus, the most rational thing for Rationality (qua idealized algorithm) to do is to precommit to strategies for P1 and P2 that maximize its expected utility given this uncertainty.
That is, let EU₁(s, t) be the expected utility to P1 if P1 implements strategy s and P2 implements strategy t. Define EU₂(s, t) similarly. Then Rationality wants to precommit the players to the respective strategies s and t that maximize
p EU₁(s, t) + q EU₂(s, t),
where p (respectively, q) is the measure of the copies of Rationality that end up in P1 (respectively, P2).
Assuming that p = q, Rationality will therefore have P1 choose C (with probability 1) and have P2 choose Y (with probability 1).
That’s all well and good, but will the players actually act that way? After all, they are stipulated to care only about themselves, so P2 in particular will not want to act according to this selfless strategy.
Yes, but this selfish desire on P2′s part is not built into P2′s Rationality subroutine (call it R), because Rationality is universal. P2′s selfish desires must be implemented elsewhere, outside of R. To be sure, P2 is free to feed the information that it is P2 to R, but R won’t do anything with this information, because R is already precommitted to a strategy for the reasons given above.
And since the players are given to be rational, they are forced to act according to the strategies pre-selected by their Rationality subroutines, despite their wishes to the contrary. Therefore, they will in fact act as Rationality determined.
[If this comment has any point, it is that there is a strong tension, if not a contradiction, between the idea of rationality as a universally valid mode of reasoning, on the one hand, and the idea of rational agents whose revealed preferences are selfish, on the other.]
Error: Adding values from different utility functions.
See this comment.
[Resuming my tongue-in-cheek argument...]
It is true that adding different utility functions is in general an error. However, for agents bound to follow Rationality (and Rationality alone), the different utility functions are best thought of as the same utility function conditioned on different hypotheses, where the different hypotheses look like “The utility to P2 turns out to be what really matters”.
After all, if the agents are making their decisions on the basis of Rationality alone, then Rationality alone must have a utility function. Since Rationality is universal, the utility function must be universal. What alternative does Rationality have, given the constraints of the problem, other than a weighted sum of the utility functions of the different individuals who might turn out to matter?
“Rationality” seems to give different answer to the same problem posed with different affine transformations of the players’ utility functions.
[Still arguing with tongue in cheek...]
That’s where the measures p and q come in.