Let’s say the current policy has a 90% chance of cooperating. Then, what action results in the highest expected reward for player 1 (and in turn, gets reinforced the most on average)? Player 1 sampling defect leads to a higher reward for player 1 whether or not player 2 samples cooperate (strategic dominance), and there’s a 90% chance of player 2 sampling cooperate regardless of player 1′s action because the policy is fixed (i.e., player 1 cooperating is no evidence of player 2 cooperating, so it’s not the case that reward tends to be higher for player 1 when player 1 cooperates as a result of player 2 tending to cooperate more in those cases). Therefore, defect actions tend to get reinforced more.
I think the thing I was missing was that in a typical RL implementation you should expect the two copies of the same policy to use different seeds, where I was imagining it as a “logical twin PD” situation where your actions are actually evidence for your twins’ actions.
Let’s say the current policy has a 90% chance of cooperating. Then, what action results in the highest expected reward for player 1 (and in turn, gets reinforced the most on average)? Player 1 sampling
defectleads to a higher reward for player 1 whether or not player 2 samplescooperate(strategic dominance), and there’s a 90% chance of player 2 samplingcooperateregardless of player 1′s action because the policy is fixed (i.e., player 1 cooperating is no evidence of player 2 cooperating, so it’s not the case that reward tends to be higher for player 1 when player 1 cooperates as a result of player 2 tending to cooperate more in those cases). Therefore,defectactions tend to get reinforced more.I think the thing I was missing was that in a typical RL implementation you should expect the two copies of the same policy to use different seeds, where I was imagining it as a “logical twin PD” situation where your actions are actually evidence for your twins’ actions.