A failure mode might also be that the SaaS people are assuming the other players are not superrational. In that case a superrational player should also defect.
Without having put much thought into it, I believe (adult) humans cooperating via this mechanism is in general very unlikely. Agents cooperating relies on all agents coming to the same (or sufficiently similar?) conclusion regarding the payoff matrix and the nature of the other agents. So in human terms, this relies on everyones ability to reason correctly about the problem and everyone elses behavior AND everyone having the right information. I don’t think that happens very often, if at all. The “everyone predicting each others behavior correctly” part seems especially unlikely to me. Also slightly different (predicted) information (e.g. AGI timelines in our case) can yield very different payoff matrices?
A failure mode might also be that the SaaS people are assuming the other players are not superrational. In that case a superrational player should also defect.
Without having put much thought into it, I believe (adult) humans cooperating via this mechanism is in general very unlikely. Agents cooperating relies on all agents coming to the same (or sufficiently similar?) conclusion regarding the payoff matrix and the nature of the other agents. So in human terms, this relies on everyones ability to reason correctly about the problem and everyone elses behavior AND everyone having the right information. I don’t think that happens very often, if at all. The “everyone predicting each others behavior correctly” part seems especially unlikely to me. Also slightly different (predicted) information (e.g. AGI timelines in our case) can yield very different payoff matrices?