Hofstadter’s Superrationality
Possibly the main and original inspiration for Yudkowsky’s various musings on what advanced game theories should do (eg. cooperate in the Prisoner’s Dilemma) is a set of essays penned by Douglas Hofstadter (of Godel, Escher, Bach) 1983. Unfortunately, they were not online and only available as part of a dead-tree collection. This is unfortunate. Fortunately the collection is available through the usual pirates as a scan, and I took the liberty of transcribing by hand the relevant essays with images, correcting errors, annotating with links, etc: http://www.gwern.net/docs/1985-hofstadter
The 3 essays:
discuss the Prisoner’s dilemma, the misfortune of defection, what sort of cooperative reasoning would maximize returns in a souped-up Prisoner’s dilemma, and then offers a public contest
then we learn the results of the contest, and a discussion of ecology and the tragedy of the commons
finally, Hofstadter gives an extended parable about cooperation in the face of nuclear warfare; it is fortunate for us that it applies to most existential threats as well
I hope you find them educational. I am not 100% confident of the math transcriptions since the original ebook messed some of them up; if you find any apparent mistakes or typos, please leave comments.
- 's comment on Intergenerational trauma impeding cooperative existential safety efforts by (4 Jun 2022 7:07 UTC; 51 points)
- Gems from the Wiki: Acausal Trade by (13 Sep 2020 0:23 UTC; 42 points)
- 's comment on newcomb’s altruism by (21 Dec 2012 10:49 UTC; 3 points)
- 's comment on Open Thread May 30 - June 5, 2016 by (31 May 2016 20:53 UTC; 2 points)
- 's comment on The True Prisoner’s Dilemma by (7 Aug 2012 5:40 UTC; 2 points)
- 's comment on [Link] Reddit, help me find some peace I’m dying young by (18 Aug 2012 16:24 UTC; 1 point)
The first essay is by far the best introduction to TDT-like reasoning that I’ve ever read. In fact this paragraph sums up the whole informal part of the idea:
Hofstadter’s comparison of “choice” and “reasoning” is getting at the idea that people have decision routines rooted in physics, which can themselves be reasoned about, including reasoning that they are similar to one’s own. I think this is really the core insight of the TDT idea.
And then the one-sentence:
— Heraclitus
Wow. The whole acausal thing feels a lot less mysterious now. Thanks.
Did you miss this comment or did you perceive it to be less intelligible?
I didn’t perceive it as talking about the same thing. Randaly’s comment defined what acausal means (which I already knew). alexflint’s comment explained why acausal influences aren’t mysterious.
“as I am no different from anyone else as far as rational thinking is concerned” is the part that bothers me about this. This approach makes sense to me in the context of clones or Tegmark duplicates or ideal reasoning agents, sure, but in the context of actual other human beings? Not a chance. And I think the results of Hoftstadter’s experiments proved that trusting other humans in this sense wouldn’t work.
I keep thinking that this is one of the big reasons identity and group politics are so prevalent. It helps answer the question “is this person sufficiently like me?”.
Can you expand what TDT represents in “TDT-like reasoning” and “TDT idea”? [I’m new here, and this is the first time I’ve seen this abbreviation on the site.]
Timeless Decision Theory.
Thank you. :)
Hofstadter presents the problems of cooperation in a context of mainstream risks — risks that public intellectuals and scientists of the time agreed were legitimate topics for discussion, and which members of the news-watching public would have heard of — such as nuclear war and environmental pollution. Yudkowsky presents these problems in a context which features exotic risks — risks that most folks have not heard of outside of science fiction; and particularly ones that deal with agents with nonhuman drives, such as Unfriendly AIs, paperclip maximizers, baby-eating aliens, and so on.
This seems like a matter of literary genre. The math of the Prisoner’s Dilemma works the same regardless of whether you’re worried about cooperating with the Kremlin or an alien. But it probably has some consequences for how people think of the subject. Someone exposed to superrationality/x-rationality ideas via Less Wrong might get the erroneous impression that they are somehow fundamentally linked to exotic risks.
On the other hand, bringing exotic agents into the discussion takes a bunch of cheap answers off the table — “Oh, humans are fundamentally cooperative; we all share the same values deep down; all we need to do is hug it out, trust each other, and do the right thing.” The math works even when you don’t share the same values, which is a pretty significant point.
If I’d been one of the participants on Hofstadter’s original game, I’d have answered him thusly:
“I know where you’re going with this experiment— you want all of us to realize that our reasoning is roughly symmetrical, and that it’s better if we all cooperate than if we all defect. And if I were playing against a bunch of copies of myself, then I’d cooperate without hesitation.
However, if I were playing against a bunch of traditional game theorists, then the sensible thing is to defect, since I know that they’re not going to reason by this line of thought, and so the symmetry is broken. Even if I were playing against a bunch of people who’d cooperate because they think that’s more moral, I ought to defect (if I’m acting according to my own self-interest), because they’re not thinking in these terms either.
So what I really need to do is to make my best guess about how many of the participants are thinking in this reflexive sort of way, and how many are basing their decisions on completely different lines of thought. And then my choice would in effect be choosing for that block of people and not for the rest, and so I’d need to make my best guess whether it’s better for me if I (and the rest of that block) choose to cooperate or if we choose to defect. That depends on how large that block is, how many of the others I expect to cooperate vs. defect, and on the payoff matrix.”
At the time he wrote it, the correct choice would have been to defect, because as Hofstadter noted, none of his friends (as brilliant as they were) took anything like that reflexive line of thought. If it were done now, among a group of Less Wrong veterans, I might be convinced to cooperate.
I would advocate the opposite: Imagine you have never thought about Newcomb-like scenarios before. Therefore, you also don’t know how others would act in such problems. Now, you come up with this interesting line of thought about determining the others’ choices or correlating with them. Because you are the only data point, your decision should give you a lot of evidence about what others might do, i.e. about whether they will come up with the idea at all and behave in abidance with it.
Now, contrast this with playing the game today. You may have already read studies showing that most philosophers use CDT, that most people one-box in Newcomb’s problem, that LWers tend to cooperate. If anything, your decision now gives you less information about what the others will do.
Huh. I finally understand the “logic” that has been espoused in HPMoR, ch 33 “”Precisely,” said Harry Potter, his face now turning serious. “We are faced with a true Prisoner’s Dilemma...”
What this reminds me of is the logistic equation: dx/dt = x(1-x).
This simple system has two equilibrium points: x = 0, and x = 1. x=0 is unstable—that is, any perturbation will cause the system to veer away from that equilibrium point. x=1 is stable, and any perturbations return to that equilbrium.
Hofstadter says that superrationalists should decide to pick the x=0 (unstable) equilibrium—i.e., cooperate. But any deviation from superrationality, however slight, will cause the equilibrium to collapse into the all-defect equilibrium.
“Reverbrating doubt”, I believe Hofstadter’s term is.
I feel like that’s not the way it should work in a worked-out theory. Maybe I or someone else will write a post about this someday.
This comment by Vladimir Nesov is relevant.
Yes. This second order issue where publicly (or predictably) pre-committing to “not negotiate with terrorists” doesn’t come through in many discussions.
Translating the insight through to Hofstadter’s parable of the tolling bell, the lack of any clear motive on the part of the postcard-demanding demon’s becomes much more vivid. Why didn’t Hofstadter give the demons a realistic motivation? Modeling the demon’s as having an inscrutably weird appreciation for postcards, the apathy of the townsfolk (who largely refusing to capitulate and spend large amounts of time writing postcards) could be seen as a refusal to “negotiate with terrorists” and in this sense it is perhaps morally praiseworthy? If the demons could have predicted the apathy/refusal but really wanted to change the townsfolk’s behavior, perhaps they might have used their powers in some other way?
Like maybe the demons, with their magic ability to conjure chemicals and build pipes, could have magically given people sprinkler systems for their lawn when they write a certain number of postcards? :-)
This problem is very very very difficult.
Unfortunately, advocates of paralysis are usually too paralyzed to advocate paralysis, or I’d suggest we form a religion for paralysis or something.
Those links were interesting. Thanks! It is weird how one of the articles you linked to is only at three. Also, Rain’s play should be an article.