# Most Prisoner’s Dilemmas are Stag Hunts; Most Stag Hunts are Schelling Problems

I previously claimed that most apparent Prisoner’s Dilemmas are actually Stag Hunts. I now claim that they’re Schelling Pub in practice. I conclude with some lessons for fighting Moloch.

*This post turned out especially dense with inferential leaps and unexplained terminology. If you’re confused, try to ask in the comments and I’ll try to clarify.*

*Some ideas here are due to Tsvi Benson-Tilsen.*

The title of this post used to be * Most Prisoner’s Dilemmas are Stag Hunts; Most Stag Hunts are Battle of the Sexes*. I’m changing it based on this comment. “Battle of the Sexes” is a game where a male and female (let’s say Bob and Alice) want to hang out, but each of them would prefer to engage in gender-stereotyped behavior. For example, Bob wants to go to a football game, and Alice wants to go to a museum. The gender issues are distracting, and although it’s the standard,

*the game isn’t that well-known anyway,*so sticking to the standard didn’t buy me much (in terms of reader understanding).

I therefore present to you,

**the Schelling Pub Game:**

Two friends would like to meet at the pub. In order to do so, they must make the same selection of pub (making this a Schelling-point game). However, they have different preferences about which pub to meet at. For example:

Alice and Bob would both like to go to a pub this evening.

There are two pubs: the Xavier, and the Yggdrasil.

Alice likes the Xavier twice as much as the Yggdrasil.

Bob likes the Yggdrasil twice as much as the Xavier.

However, Alice and Bob also prefer to be with each other. Let’s say they like being together ten times as much as they like being apart.

Schelling Pub Game payoff matrix | |||

payoffs written alice;bob | B’s choice | ||

X | Y | ||

A’s choice | X | 20;10 | 2;2 |

Y | 1;1 | 10;20 |

The important features of this game are:

The Nash equilibria are all Pareto-optimal. There is no “individually rational agents work against each other” problem, like in prisoner’s dilemma or even stag hunt.

There are multiple equilibria, and different agents prefer different equilibria.

Thus, realistically, agents may not end up in equilibrium at all—because (in the single-shot game) they don’t know which to choose, and because (in an iterated version of the game) they may make locally sub-optimal choices in order to influence the long-run behavior of other players.

(Edited to add, based on comments:)

Here’s a summary of the central argument which, despite the lack of pictures, may be easier to understand.

Most Prisoner’s Dilemmas are actually iterated.

Iterated games are a whole different game with a different action space (because you can react to history), a different payoff matrix (because you care about future payoffs, not just the present), and a different set of equilibria.

It is characteristic of PD that players are incentivised to play away from the Pareto frontier; IE, no Pareto-optimal point is an equilibrium.

*This is not the case with iterated PD.*It is characteristic of Stag Hunt that there is a Pareto-optimal equilibrium, but there is also another equilibrium which is far from optimal.

*This is also the case with iterated PD.*So**iterated PD resembles Stag Hunt**.However, it is furthermore true of iterated PD that

*there are multiple different Pareto-optimal equilibria, which benefit different players more or less.*Also, if players don’t successfully coordinate on one of these equilibria, they can end up in a worse overall state (such as mutual defection forever, due to playing grim-trigger strategies with mutually incompatible demands).**This makes iterated PD resemble the Schelling Pub Game.**

In fact, the Folk Theorem suggests that *most *iterated games will resemble the Schelling Pub Game in this way.

In a comment on The Schelling Choice is “Rabbit”, not “Stag” I said:

In the book

The Stag Hunt,Skyrms similarly says that lots of people use Prisoner’s Dilemma to talk about social coordination, and he thinks people should often use Stag Hunt instead.

I think this is right. Most problems which initially seem like Prisoner’s Dilemma are actually Stag Hunt, because there are potential enforcement mechanisms available. The problems discussed in Meditations on Moloch are mostly Stag Hunt problems, not Prisoner’s Dilemma problems—Scott even talks about enforcement, when he describes the dystopia where everyone has to kill anyone who doesn’t enforce the terrible social norms (including the norm of enforcing).

This might initially sound like good news. Defection in Prisoner’s Dilemma is an inevitable conclusion under common decision-theoretic assumptions. Trying to escape multipolar traps with exotic decision theories might seem hopeless. On the other hand, rabbit in Stag Hunt is

notan inevitable conclusion, by any means.

Unfortunately, in reality, hunting stag is actually quite difficult.

(“The schelling choice is Rabbit, not Stag… and that really sucks!”)

Inspired by Zvi’s recent sequence on Moloch, I wanted to expand on this. These issues are important, since they determine how we think about group action problems / tragedy of the commons / multipolar traps / Moloch / all the other synonyms for the same thing.

My current claim is that most Prisoner’s Dilemmas are actually *Schelling pub games*. But let’s first review the relevance of Stag Hunt.

# Your PD Is Probably a Stag Hunt

There are several reasons why an apparent Prisoner’s Dilemma may be more of a Stag Hunt.

The game is actually an iterated game.

Reputation networks could punish defectors and reward cooperators.

There are enforceable contracts.

Players know quite a bit about how other players think (in the extreme case, players can view each other’s source code).

Each of these formal model creates a situation where players * can* get into a cooperative equilibrium. The challenge is that you can’t unilaterally decide everyone should be in the cooperative equilibrium. If you want good outcomes for yourself, you have to account for what everyone else probably does. If you think everyone is likely to be in a bad equilibrium where people punish each other for cooperating, then aligning with that equilibrium might be the best you can do! This is like hunting rabbit.

**Exercize**: is there a situation in your life, or within spitting distance, which seems like a Prisoner’s Dilemma to you, where everyone is stuck hurting each other due to bad incentives? Is it an iterated situation? Could there be reputation networks which weed out bad actors? Could contracts or contract-like mechanisms be used to encourage good behavior?

So, why do we perceive so many situations to be Prisoner’s Dilemma -like rather than Stag Hunt -like? Why does Moloch sound more like *each individual is incentivized to make it worse for everyone else* than *everyone is stuck in a bad equilibrium?*

Sarah Constantine writes:

A friend of mine speculated that, in the decades that humanity has lived under the threat of nuclear war, we’ve developed the assumption that we’re living in a world of one-shot Prisoner’s Dilemmas rather than repeated games, and lost some of the social technology associated with repeated games. Game theorists do, of course, know about iterated games and there’s some fascinating research in evolutionary game theory, but the original formalization of game theory was for the application of nuclear war, and the 101-level framing that most educated laymen hear is often that one-shot is the prototypical case and repeated games are hard to reason about without computer simulations.

To use board-game terminology, the *game* may be a Prisoner’s Dilemma, but the *metagame *can use enforcement techniques. Accounting for enforcement techniques, the game is more like a Stag Hunt, where defecting is “rabbit” and cooperating is “stag”.

# Schelling Pubs

But this is a bit informal. You don’t separately choose how to metagame and how to game; really, your iterated strategy determines what you do in individual games.

So it’s more accurate to just think of the iterated game. There are a bunch of iterated strategies which you can choose from.

The key difference between the single-shot game and the iterated game is that cooperative strategies, such as Tit for Tat (but including others), are avaliable. These strategies have the property that (1) they are equilibria—if you know the other player is playing Tit for Tat, there’s no reason for you not to; (2) if both players use them, they end up cooperating.

A key feature of Tit for Tat strategy is that if you do end up playing against a pure defector, you do almost as well as you could possibly do with them. This doesn’t sound very much like a Stag Hunt. It begins to sound like a Stag Hunt in which you can change your mind and go hunt rabbit if the other person doesn’t show up to hunt stag with you.

Sounds great, right? We can just play one of these cooperative strategies.

The problem is, there are many possible self-enforcing equilibria. Each player can threaten the other player with a *Grim Trigger* strategy: they defect forever the moment some specified condition isn’t met. This can be used to extort the other player for more than just the mutual-cooperation payoff. Here’s an illustration of possible outcomes, with the enforceable frequencies in the white area:

Alice could be extorting Bob by cooperating 2/3rds of the time, with a grim-trigger threat of never cooperating at all. Alice would then get an average payoff of 2⅓, while Bob would get an average payout of 1⅓.

In the artificial setting of Prisoner’s Dilemma, it’s easy to say that Cooperate, Cooperate is the “fair” solution, and an equilibrium like I just described is “Alice exploiting Bob”. However, real games are not so symmetric, and so it will not be so obvious what “fair” is. The purple squiggle highlights the Pareto frontier—the space of outcomes which are “efficient” in the sense that no alternative is purely better for everybody. These outcomes may not all be fair, but they all have the advantage that no “money is left on the table”—any “improvement” we could propose for those outcomes makes things worse for at least one person.

Notice that I’ve also colored areas where Bob and Alice are doing worse than payoff 1. Bob can’t enforce Alice’s cooperation while defecting more than half the time; Alice would just defect. And vice versa. All of the points within the shaded regions have this property. So not *all* Pareto-optimal solutions can be enforced.

Any point in the white region can be enforced, however. Each player could be watching the statistics of the other player’s cooperation, prepared to pull a grim-trigger if the statistics ever stray too far from the target point. This includes so-called * mutual blackmail* equilibria, in which both players cooperate with probability slightly better than zero (while threatening to never cooperate at all if the other player detectably diverges from that frequency). This idea—that ‘almost any’ outcome can be enforced—is known as the Folk Theorem in game theory.

The Schelling Pub part is that (particularly with grim-trigger enforcement) everyone has to choose the same equilibrium to enforce; otherwise everyone is stuck playing defect. You’d rather be in even a bad mutual-blackmail type equilibrium, as opposed to selecting incompatible points to enforce. Just like, in Schelling Pub, you’d prefer to meet together at any venue rather than end up at different places.

Furthermore, I would claim that *most* apparent Stag Hunts which you encounter in real life are actually schelling-pub, in the sense that there are many different stags to hunt and it isn’t immediately clear which one should be hunted. Each stag will be differently appealing to different people, so it’s difficult to establish common knowledge about which one is worth going after together.

**Exercize**: what stags aren’t you hunting with the people around you?

# Taking Pareto Improvements

Fortunately, Grim Trigger is not the *only* enforcement mechanism which can be used to build an equilibrium. Grim Trigger creates a crisis in which you’ve got to guess which equilibrium you’re in very quickly, to avoid angering the other player; and no experimentation is allowed. There are much more forgiving strategies (and contrite ones, too, which helps in a different way).

Actually, *even using Grim Trigger to enforce things*, why would you punish the other player for doing something *better for you? *There’s no motive for punishing the other player for raising their cooperation frequency.

In a scenario where you don’t know which Grim Trigger the other player is using, but you don’t think they’ll punish you for cooperating *more* than the target, a natural response is for both players to just cooperate a bunch.

So, it can be very valuable to **use enforcement mechanisms which allow for Pareto improvements.**

Taking Pareto improvements is about moving from the middle to the boundary:

(I’ve indicated the directions for Pareto improvements starting from the origin in yellow, as well as what happens in other directions; also, I drew a bunch of example Pareto improvements as black arrows to illustrate how Pareto improvements are awesome. Some of the black arrows might not be perfectly within the range of Pareto improvements, sorry about that.)

However, there’s also an argument against taking Pareto improvements. If you accept *any* Pareto improvements, you can be exploited in the sense mentioned earlier—you’ll accept any situation, so long as it’s not worse for you than where you started. So you will take some pretty poor deals. Notice that one Pareto improvement can prevent a different one—for example, if you move to (1/2, 1), then you can’t move to (1,1/2) via Pareto improvement. So you could always reject a Pareto improvement because you’re holding out for a better deal. (This is the *Schelling Pub* aspect of the situation—there are Pareto-optimal outcomes which are better or worse for different people, so, it’s hard to agree on which improvement to take.)

That’s where Cooperation between Agents with Different Notions of Fairness comes in. The idea in that post is that you don’t take *just any* Pareto improvement—you have standards of fairness—but you don’t just completely defect for less-than-perfectly-fair deals, either. What this means is that two such agents with incompatible notions of fairness can’t get all the way to the Pareto frontier, but the closer their notions of fairness are to each other, the closer they can get. And, if the notions of fairness *are* compatible, they can get all the way.

# Moloch is the Folk Theorem

Because of the Folk Theorem, *most* iterated games will have the same properties I’ve been talking about (not just iterated PD). Specifically, most iterated games will have:

**Stag-hunt-like property 1:**There is a Pareto-optimal equilibrium, but there is also an equilibrium far from Pareto-optimal.**The Schelling Pub property:**There are multiple Pareto-optimal equilibria, so that even if you’re trying to cooperate, you don’t necessarily know which one to aim for; and, different options favor different people, making it a complex negotiation even if you can discuss the problem ahead of time.

There’s a third important property which I’ve been assuming, but which doesn’t follow so directly from the Folk Theorem: **the suboptimal equilibrium is “safe”, in that you can unilaterally play that way to get some guaranteed utility.** The Pareto-optimal equilibria are not similarly safe; mistakenly playing one of them when other people don’t can be worse than the “safe” guarantee from the poor equilibrium.

A game with all three properties is like Stag Hunt with multiple stags (where you all must hunt the same stag to win, but can hunt rabbit alone for a guaranteed mediocre payoff), or Schelling Pub where you can just stay home (you’d rather stay home than go out alone).

# Lessons in Slaying Moloch

0. I didn’t even address this in this essay, but it’s worth mentioning: * not all conflicts are zero-sum.* In the introduction to the 1980 edition of

*The Strategy of Conflict*, Thomas Schelling discusses the reception of the book. He recalls that a prominent political theorist “exclaimed how much this book had done for his thinking, and as he talked with enthusiasm I tried to guess which of my sophisticated ideas in which chapters had made so much difference to him. It turned out it wasn’t any particular idea in any particular chapter. Until he read this book, he had simply not comprehended that an inherently non-zero-sum conflict could exist.”

1. In situations such as iterated games, * there’s no in-principle pull toward defection.* Prisoner’s Dilemma seems paradoxical when we first learn of it (at least, it seemed so to me) because we are not accustomed to such a harsh divide between individual incentives and the common good. But perhaps, as Sarah Constantine speculated in Don’t Shoot the Messenger, modern game theory and economics have conditioned us to be used to this conflict due to their emphasis on single-shot interactions. As a result, Moloch comes to sound like an inevitable gravity, pulling everything downwards. This is not necessarily the case.

2. Instead, * most collective action problems are bargaining problems*. If a solution can be agreed upon, we can generally use weak enforcement mechanisms (social norms) or strong enforcement (centralized governmental enforcement) to carry it out. But, agreeing about the solution may not be easy. The more parties involved, the more difficult.

3. * Try to keep a path open toward better solutions.* Since wide adoption of a particular solution can be such an important problem, there’s a tendency to treat alternative solutions as the enemy. This bars the way to further progress. (One could loosely characterize this as the difference between religious doctrine and democratic law; religious doctrine trades away the ability to improve in favor of the more powerful consensus-reaching technology of immutable universal law. But of course this oversimplifies things somewhat.) Keeping a path open for improvements is hard, partly because it can create exploitability. But it keeps us from getting stuck in a poor equilibrium.

- Responses to apparent rationalist confusions about game / decision theory by 30 Aug 2023 22:02 UTC; 140 points) (
- Book Launch: “The Carving of Reality,” Best of LessWrong vol. III by 16 Aug 2023 23:52 UTC; 131 points) (
- Voting Results for the 2020 Review by 2 Feb 2022 18:37 UTC; 108 points) (
- The Prototypical Negotiation Game by 20 Feb 2021 21:33 UTC; 96 points) (
- Prizes for the 2020 Review by 20 Feb 2022 21:07 UTC; 94 points) (
- Implications of Civilizational Inadequacy (reviewing mazes/simulacra/etc) by 21 Jan 2022 1:45 UTC; 93 points) (
- Norm Innovation and Theory of Mind by 18 Sep 2021 21:38 UTC; 91 points) (
- Iterated Trust Kickstarters by 20 Apr 2021 3:18 UTC; 74 points) (
- 2020 Review Article by 14 Jan 2022 4:58 UTC; 74 points) (
- Quantitative cruxes in Alignment by 2 Jul 2023 20:38 UTC; 19 points) (
- Infra-Bayesian haggling by 20 May 2024 12:23 UTC; 18 points) (
- 12 Feb 2022 22:28 UTC; 6 points) 's comment on Voting Results for the 2020 Review by (
- 13 May 2023 4:32 UTC; 4 points) 's comment on Thoughts on LessWrong norms, the Art of Discourse, and moderator mandate by (
- 5 Dec 2020 20:25 UTC; 4 points) 's comment on The Schelling Choice is “Rabbit”, not “Stag” by (
- 15 Sep 2020 7:22 UTC; 2 points) 's comment on Are there non-AI projects focused on defeating Moloch globally? by (
- 5 Sep 2021 9:08 UTC; 2 points) 's comment on Acausal Trade and the Ultimatum Game by (

A short note to start the review that the author isn’t happy with how it is communicated. I agree it could be clearer and this is the reason I’m scoring this 4 instead of 9. The actual content seems very useful to me.

AllAmericanBreakfast has already reviewed this from a theoretical point of view but I wanted to look at it from a practical standpoint.

***

To test whether the conclusions of this post were true in practice I decided to take 5 examples from the Wikipedia page on the Prisoner’s dilemma and see if they were better modeled by Stag Hunt or Schelling Pub:

Climate negotiations

Relationships

Marketing

Doping in sport

Cold war nuclear arms race

Detailed analysis of each is at the bottom of the review.

Of these 5, 3 (Climate, Relationships, Arms race) seem to me to be very well modeled by Schelling Pub.

Due to the constraints on communication allowed between rival companies it is difficult to see marketing (where more advertising = defect) as a Schelling Pub game. There probably is an underlying structure which looks a bit like Schelling Pub but it is very hard to move between Nash Equilibria. As a result I would say that Prisoner’s Dilemma is a more natural model for marketing.

The choice of whether to dope in sport is probably best modeled as a Prisoner’s dilemma with an enforcing authority which punishes defection. As a result, I don’t think any of the 3 games are a particularly good model for any individual’s choice. However, negotiations on setting up the enforcing authority and the rules under which it operates are more like Schelling Pub. Originally I thought this should maybe count as half a point for the post but thinking about it further I would say this is actually a very strong example of what the post is talking about – if your individual choice looks like a Prisoner’s Dilemma then look for ways to make it into a Schelling Pub. If this involves setting up a central enforcement agency then negotiate to make that happen.

So I’m scoring it 4 out of 5 Prisoners Dilemmas examined are better modeled as Schelling Pubs, which is in line with the “most” claims of the post title.

The example which was least like Schelling Pub was the one where communication was difficult/impossible due to external rules. I think the value of communication is implicit in the post but it would be helpful to have it spelled out explicitly.

One other thing which might be useful from a practical point is that things which don’t initially seem iterated may be able to be iterated if you split them into smaller tasks. You don’t have to reduce your nuclear arsenal or decarbonise all at once, you can do a little bit and then check that the others have done the same before continuing. This seems obvious on a national level but maybe not so obvious on a personal level.

***

(Read beyond here only if you’re interested in more detail of the examples—it doesn’t add to the conclusions)

Below is my analysis of the 5 items chosen from the Prisoner’s dilemmas example list on Wikipedia. In discussing Stag Hunts (SH) I use the post’s list of 4 items which might make something more like a SH than a Prisoner’s Dilemma (PD).

Climate negotiationsPD shaped

Each country benefits from stable climate

Each would prefer that they put in minimum effort to achieve it

SH shaped?

Iteration: Yes—it is iterated over years and agreements

Reputation: It seems like yes, although I’m not sure how this works out in practice

Enforecable contracts: Not really

Superrationality: Possibly

SP shaped?

It seems yes.

As an example, recall the coal amendment from the Glasgow talks

This is an amendment which favours some countries over others and means that those disfavoured will put in relatively more effort for a given amount of CO2 reduction compared to the original wording.

It seems obvious that the actual agreement probably isn’t on the Pareto frontier

I think the final figure from the post gives a good mental model of what is going on

RelationshipsPD shaped

In theory, for an individual action, you’re better off if you get your own way over your partner

SH shaped?

Iteration: Yes (+ your partner can just leave)

Reputation: Very important in small communities, less so in large ones

Enforceable contracts: Sometimes (prenup)

Superrationality: Potentially yes but not required

It will be better to hunt stag together just from iteration

Abusive relationships are not SH shaped as the partner can’t or won’t leave, especially when there’s no reputational effect that the abusive partner cares about

SP shaped?

Ideally.

MarketingPD shaped

2 rival companies with equally effective marketing departments are in a roughly PD shaped game (assuming customer pool is fixed size)

If one spends money on advertising (defect) then the other is disadvantaged if they don’t

But both would be better off if neither advertised

Often the customer pool is not of fixed size which would mean that this may not really be a PD in real life

It is important here to note that collusion between companies is generally forbidden so communication is not allowed

SH shaped?

Iteration: Yes.

Reputation: Not really – there aren’t negative reputation effects to advertising

Enforceable contracts: No

Modelling other players: Yes but they are (accurately) modelling each other as playing defect

In theory the businesses could get into C-C but advertising is so ingrained as the default choice that this would be hard

Possibly there’s an availability heuristic problem here – I’ll obviously remember the examples of industries which are stuck in D-D as I constantly see their adverts (Pepsi vs Coke, tech companies, supermarkets).

I tried to think of industries that aren’t advertising much but I’m drawing a blank.

SP shaped?

In theory maybe, however the lack of communication makes it extremely hard for companies to change between solutions

Doping in sportPD shaped

Using drugs gives you an edge but has a price potential price with medical dangers

There is a potential difference from “standard” PD in that doping is disadvantageous to a more able athlete if the less able athlete is not doping (which would make this an alibi game).

SH shaped?

Iteration: Yes (although results of previous iterations are not very legible)

Reputation: Yes (again, legibility problems)

Enforceable contract: Yes – drugs testing and bans

Superrationality: No

The enforceable contract is probably the biggest effect here – just relying on iteration and reputation would be insufficient

SP shaped?

Because the solution is an enforceable contract the decision is not very SP shaped on an individual basis

Negotiating what the contract should be is SP shaped

What makes a medical exemption ok?

How much inhaler is an ok amount?

Cold war nuclear arms racePD shaped

Making more nukes is equivalent to defect

SH shaped?

Iteration: Yes, treaty followed treaty. Inspections were allowed to verify adherence.

Reputation: A bit but probably not in a relevant way

Enforceable contract: Not really – if one country reneged then it’s not like the other country could sue

Superrationality: Yes. It seems both players realised D-D was terrible and wanted to play C-C. They couldn’t rely on just this so iteration was very important.

SP shaped?

Deciding the terms of the agreement is SP shaped

What size arsenal is better for which country?

Which particular weapons are better for which country?

The goal of this post is to help us understand the similarities and differences between several different games, and to improve our intuitions about which game is the right default assumption when modeling real-world outcomes.

My main objective with this review is to check the game theoretic claims, identify the points at which this post makes empirical assertions, and see if there are any worrisome oversights or gaps. Most of my fact-checking will just be resorting to Wikipedia.

Let’s start with definitions of two key concepts.

Pareto-optimal:One dimension cannot improve without a second worsening.Nash equilibrium:No player can do better by unilaterally changing their strategy.Here’s the payoff matrix from the one-shot Prisoner’s Dilemma and how it relates to these key concepts.

This article outlines three possible relationships between Pareto-optimality and Nash equilibrium.

There are no Pareto-optimal Nash equilibria.

There is a single Pareto-optimal Nash equilibrium, and another equilibrium that is not Pareto-optimal.

There are multiple Pareto-optimal Nash equilibria, which benefit different players to different extents.

The author attempts to argue which of these arrangements best describes the world we live in, and makes the best default assumption when interpreting real-world situations as games. The claim is that real-world situations most often resemble iterated PDs, which have multiple Pareto-optimal Nash equilibria benefitting different players to different extents. I will attempt to show that the author’s conclusion only applies when modeling superrational entities, or entities with an unbounded lifespan, and give some examples where this might be relevant.

Iterated Prisoner’s Dilemma is a little more complex than the author states. If the players know how many turns the game will be played for, or if the game has a known upper limit of turns, the Nash equilibrium is always to defect. However, if the players are superrational, meaning that they not only are perfectly rational but also assume all other players are too and that superrational players always converge on the same strategy, then they’ll always cooperate.

As such, the Nash equilibrium for rational, but not superrational, players for games with fixed or upper-bounded N is the same as for the single-shot game. In real life, any game played between human beings that takes a non-zero amount of time has an upper bound on the number of turns, given that we currently must expect ourselves to die. Therefore, game theory suggests that the Nash equilibrium strategy for all iterated Prisoner’s Dilemmas between rational players is defect/defect. Therefore, the claims about iterated PD in steps 2-5 in the author’s argument summary only seems to hold if we are talking about non-human entities with unbounded life expectancies, or if humans are modeled as superrational agents.

Let’s gesture at some plausible but extremely speculative real-world examples of how games with an unbounded upper limit of turns or superrationality might be reasonable models for human games.

Social entities, such as governments, corporations, and cultures, if they can be modeled as agents, could be seen as having unbounded lifespans. When we make strict game-theoretic arguments, we can be equally strict in mathematical assumptions about other facets of reality. If understanding of physics is imperfect, and if there is a non zero possibility of some continuity of agents not just into the distant future, but infinitely into time, then they could be modeled as having unbounded lifespans. If this holds, and if social entities are causally responsible for the way that most games unfold, then this may rescue the argument.

A second angle on this idea is that humans may have a psychological tendency to interpret long periods of time as equivalent to infinite in length, analogously to how people intuitively round low probabilities to 0 and high probabilities to 1. This suggests that studies investigating strategies empirically chosen by people playing iterated PDs in the lab would increasingly approach optimal strategies for unbounded iterated PDs as the turn count increases. A complicating factor here is that we must disambiguate whether the assumption is that humans “round”

long lengthsoftimeorlargenumbers of turnsto infinity.Outcomes for superrational agents are better than those for rational agents in the turn-bounded iterated PD. Imagine a large set of agents, each playing a randomly-chosen strategy. Some of those strategies may match the superrational strategy in iterated PD. If they face selection pressure over time, with some spontaneous generation of new agents, agents playing the superrational strategy may eventually dominate the population of agents. Conceivably, humans, and our social ancestor species, may have been genetically hardwired via group selection to adopt superrational strategies by instinct. This aligns with the psychological explanation.

Edit: Vanessa Kosoy pointed out below that iterated PDs with a finite but unknown number of iterations, or slightly noisy agents, can have Nash equilibria involving cooperation. We therefore don’t need to resort to such exotic explanations as I’ve offered here to explain how abramdemski’s arguments 2-5 hold, and we don’t need to “trick ourselves into cooperation” in such scenarios.If this argument holds water, how does it affect the original agenda of this article, which was to inform our intuitions about how to model real-world games? It suggests that these twin questions, of psychological “rounding” and a possible group-selection account of how this might have evolved, would be important to investigate to increase our confidence in this heuristic. If true, it also suggests vulnerabilities in normal human approaches to games. Our ability to cooperate, under this hypothesis, depends on our ability to trick ourselves into cooperation by conveniently ignoring the inevitable end of our games.

The local argument made by this post needn’t be true for its conclusion to be true, and if the post seems plausible because it aligns with our real-world experience, we might want to appreciate the article for the conclusion it articulates as well as the argument it makes in support of that conclusion. The conclusion is that in most situations, we have multiple Pareto-optimal Nash equilibria, favoring different agents. Colloquially, many human problems are about fairness and resource allocation, and the threats and strategies people use to steer negotiation toward the outcome that favors them the most, while still achieving a fundamentally cooperative outcome.

This seems to me like an articulate, usefully predictive, simple, and realistic depiction of an enormous number of fundamental challenges in human organization. Although I don’t think that the original post’s game-theoretic argument is airtight, I think its psychological and sociological plausibility in conjunction with a tweaked game-theoretic argument makes it worthwhile and interesting. I also appreciate the care the author took to summarize, update, and respond to comments. Pointing out similarities and differences in the relationship between Nash equilibria and Pareto optimality in the various games also helped me understand them better, which I appreciate.

Cooperation can be a Nash equilibrium in the IPD if you have a finite but

unknownnumber of iterations (e.g. geometrically distributed). Also, if the number of iterations is known but very large, cooperating becomes an ϵ-Nash equilibrium for small ϵ (if we normalize utility by its maximal value), so agents which are not superrational but a little noisy can still converge there (and, agents are sometimes noisy by design in order to facilitate exploration).Thank you for pointing this out. Here’s a source for the first claim.

And here’s a source that at least provides a starting point for the second claim about ϵ-Nash equilibria.

I like this, in the sense that it’s provoking fascinating thoughts and makes me want to talk with the author about it further. As a communication of a particular concept? I’m kinda having a hard time following what the intent is.