I wouldn’t really call ZD agents “endlessly escalating”, and that one seems like an exploitative ZD agent. Maybe something like “line-toeing” agents would work better? Or just call them exploitative.
P.S. If anyone felt Zvi’s post confused them because they vaguely remembered that co-operation sort of pops out of sensible decision theories, I’d recommend reading the literature on “zero determinant” agents (they can enforce linear relationships between their and their opponents payoffs in iterated prisoners’ dilemma). I think these sorts of agents are more of a problem in CDT? (or EDT? I don’t know, I’m tired). Anyway, herearesomegoodpapers on the topic, even though Zvi basically goes over the main results in the post anyway.
Also interesting is that ZD agents do better by exploiting (their score dominates opponents) humans in the manner described in the papers if you provide long time scales, and tell the human their opponent is a robot. Humans will go along with this, as the theory says, but not if they think their opponent is human. In assymetric situationsbetween humans, extortion tends to be pretty frequent their opponent is human.
I wouldn’t really call ZD agents “endlessly escalating”, and that one seems like an exploitative ZD agent. Maybe something like “line-toeing” agents would work better? Or just call them exploitative.
P.S. If anyone felt Zvi’s post confused them because they vaguely remembered that co-operation sort of pops out of sensible decision theories, I’d recommend reading the literature on “zero determinant” agents (they can enforce linear relationships between their and their opponents payoffs in iterated prisoners’ dilemma). I think these sorts of agents are more of a problem in CDT? (or EDT? I don’t know, I’m tired). Anyway, here are some good papers on the topic, even though Zvi basically goes over the main results in the post anyway.
Also interesting is that ZD agents do better by exploiting (their score dominates opponents) humans in the manner described in the papers if you provide long time scales, and tell the human their opponent is a robot. Humans will go along with this, as the theory says, but not if they think their opponent is human. In assymetric situations between humans, extortion tends to be pretty frequent their opponent is human.