Imagine reaching into an urn that contains seventy white balls and thirty red ones, and plucking out ten mystery balls.
Perhaps three of the ten balls will be red, and you’ll correctly guess how many red balls total were in the urn. Or perhaps you’ll happen to grab four red balls, or some other number. Then you’ll probably get the total number wrong.
This random error is the cost of incomplete knowledge, and as errors go, it’s not so bad. Your estimates won’t be incorrect on average, and the more you learn, the smaller your error will tend to be.
On the other hand, suppose that the white balls are heavier, and sink to the bottom of the urn. Then your sample may be unrepresentative in a consistent direction.
That kind of error is called “statistical bias.” When your method of learning about the world is biased, learning more may not help. Acquiring more data can even consistently worsen a biased prediction.
If you’re used to holding knowledge and inquiry in high esteem, this is a scary prospect. If we want to be sure that learning more will help us, rather than making us worse off than we were before, we need to discover and correct for biases in our data.
The idea of cognitive bias in psychology works in an analogous way. A cognitive bias is a systematic error in how we think, as opposed to a random error or one that’s merely caused by our ignorance. Whereas statistical bias skews a sample so that it less closely resembles a larger population, cognitive biases skew our thinking so that it less accurately tracks the truth (or less reliably serves our other goals).
Maybe you have an optimism bias, and you find out that the red balls can be used to treat a rare tropical disease besetting your brother, and you end up overestimating how many red balls the urn contains because you wish the balls were mostly red.
Like statistical biases, cognitive biases can distort our view of reality, they can’t always be fixed by just gathering more data, and their effects can add up over time. But when the miscalibrated measuring instrument you’re trying to fix is you, debiasing is a unique challenge.
Still, this is an obvious place to start. For if you can’t trust your brain, how can you trust anything else?
Noticing Bias
Imagine meeting someone for the first time, and knowing nothing about them except that they’re shy.
Question: Is it more likely that this person is a librarian, or a salesperson?
Most people answer “librarian.” Which is a mistake: shy salespeople are much more common than shy librarians, because salespeople in general are much more common than librarians—seventy-five times as common, in the United States.¹
This is base rate neglect: grounding one’s judgments in how well sets of characteristics feel like they fit together, and neglecting how common each characteristic is in the population at large.² Another example of a cognitive bias is the sunk cost fallacy—people’s tendency to feel committed to things they’ve spent resources on in the past, when they should be cutting their losses and moving on.
Knowing about these biases, unfortunately, doesn’t make you immune to them. It doesn’t even mean you’ll be able to notice them in action.
In a study of bias blindness, experimental subjects predicted that they would have a harder time neutrally evaluating the quality of paintings if they knew the paintings were by famous artists. And indeed, these subjects exhibited the very bias they had predicted when the experimenters later tested their prediction. When asked afterward, however, the very same subjects claimed that their assessments of the paintings had been objective and unaffected by the bias.³
Even when we correctly identify others’ biases, we exhibit a bias blind spot when it comes to our own flaws.⁴ Failing to detect any “biased-feeling thoughts” when we introspect, we draw the conclusion that we must just be less biased than everyone else.⁵
Yet it is possible to recognize and overcome biases. It’s just not trivial. It’s known that subjects can reduce base rate neglect, for example, by thinking of probabilities as frequencies of objects or events.
The approach to debiasing in this book is to communicate a systematic understanding of why good reasoning works, and of how the brain falls short of it. To the extent this volume does its job, its approach can be compared to the one described in Serfas (2010), who notes that “years of financially related work experience” didn’t affect people’s susceptibility to the sunk cost bias, whereas “the number of accounting courses attended” did help.
As a consequence, it might be necessary to distinguish between experience and expertise, with expertise meaning “the development of a schematic principle that involves conceptual understanding of the problem,” which in turn enables the decision maker to recognize particular biases. However, using expertise as countermeasure requires more than just being familiar with the situational content or being an expert in a particular domain. It requires that one fully understand the underlying rationale of the respective bias, is able to spot it in the particular setting, and also has the appropriate tools at hand to counteract the bias.⁶
The goal of this book is to lay the groundwork for creating rationality “expertise.” That means acquiring a deep understanding of the structure of a very general problem: human bias, self-deception, and the thousand paths by which sophisticated thought can defeat itself.
A Word About This Text
Map and Territory began its life as a series of essays by decision theorist Eliezer Yudkowsky, published between 2006 and 2009 on the economics blog Overcoming Bias and its spin-off community blog Less Wrong. Thematically linked essays were grouped together in “sequences,” and thematically linked sequences were grouped into books. Map and Territory is the first of six such books, with the series as a whole going by the name Rationality: From AI to Zombies.⁷
In style, this series run the gamut from “lively textbook” to “compendium of vignettes” to “riotous manifesto,” and the content is correspondingly varied. The resultant rationality primer is frequently personal and irreverent—drawing, for example, from Yudkowsky’s experiences with his Orthodox Jewish mother (a psychiatrist) and father (a physicist), and from conversations on chat rooms and mailing lists. Readers who are familiar with Yudkowsky from Harry Potter and the Methods of Rationality, his science-oriented take-off of J.K. Rowling’s Harry Potter books, will recognize the same iconoclasm, and many of the same themes.
The philosopher Alfred Korzybski once wrote: “A map is not the territory it represents, but, if correct, it has a similar structure to the territory, which accounts for its usefulness.” And what can be said of maps here, as Korzybski noted, can also be said of beliefs, and assertions, and words.
“The map is not the territory.” This deceptively simple claim is the organizing idea behind this book, and behind the four sequences of essays collected here: Predictably Wrong, which concerns the systematic ways our beliefs fail to map the real world; Fake Beliefs, on what makes a belief a “map” in the first place; Noticing Confusion, on how this world-mapping thing our brains do actually works; and Mysterious Answers, which collides these points together. The book then concludes with “The Simple Truth,” a stand-alone dialogue on the idea of truth itself.
Humans aren’t rational; but, as behavioral economist Dan Ariely notes, we’re predictably irrational. There are patterns to how we screw up. And there are patterns to how we behave when we don’t screw up. Both admit of fuller understanding, and with it, the hope of leaning on that understanding to build a better future for ourselves.
¹ Wayne Weiten, Psychology: Themes and Variations, Briefer Version, Eighth Edition (Cengage Learning, 2010).
² Richards J. Heuer, Psychology of Intelligence Analysis (Center for the Study of Intelligence, Central Intelligence Agency, 1999) .
³ Katherine Hansen et al., “People Claim Objectivity After Knowingly Using Biased Strategies,” Personality and Social Psychology Bulletin 40, no. 6 (2014): 691–699 .
⁴ Emily Pronin, Daniel Y. Lin, and Lee Ross, “The Bias Blind Spot: Perceptions of Bias in Self versus Others,” Personality and Social Psychology Bulletin 28, no. 3 (2002): 369–381 .
⁵ Joyce Ehrlinger, Thomas Gilovich, and Lee Ross, “Peering Into the Bias Blind Spot: People’s Assessments of Bias in Themselves and Others,” Personality and Social Psychology Bulletin 31, no. 5 (2005): 680–692.
⁶ Sebastian Serfas, Cognitive Biases in the Capital Investment Context: Theoretical Considerations and Empirical Experiments on Violations of Normative Rationality (Springer, 2010).
⁷ The first edition of Rationality: From AI to Zombies was released as a single sprawling ebook, before the series was edited and split up into separate volumes. The full book can also be found on http://lesswrong.com/rationality.
Biases: An Introduction
Imagine reaching into an urn that contains seventy white balls and thirty red ones, and plucking out ten mystery balls.
Perhaps three of the ten balls will be red, and you’ll correctly guess how many red balls total were in the urn. Or perhaps you’ll happen to grab four red balls, or some other number. Then you’ll probably get the total number wrong.
This random error is the cost of incomplete knowledge, and as errors go, it’s not so bad. Your estimates won’t be incorrect on average, and the more you learn, the smaller your error will tend to be.
On the other hand, suppose that the white balls are heavier, and sink to the bottom of the urn. Then your sample may be unrepresentative in a consistent direction.
That kind of error is called “statistical bias.” When your method of learning about the world is biased, learning more may not help. Acquiring more data can even consistently worsen a biased prediction.
If you’re used to holding knowledge and inquiry in high esteem, this is a scary prospect. If we want to be sure that learning more will help us, rather than making us worse off than we were before, we need to discover and correct for biases in our data.
The idea of cognitive bias in psychology works in an analogous way. A cognitive bias is a systematic error in how we think, as opposed to a random error or one that’s merely caused by our ignorance. Whereas statistical bias skews a sample so that it less closely resembles a larger population, cognitive biases skew our thinking so that it less accurately tracks the truth (or less reliably serves our other goals).
Maybe you have an optimism bias, and you find out that the red balls can be used to treat a rare tropical disease besetting your brother, and you end up overestimating how many red balls the urn contains because you wish the balls were mostly red.
Like statistical biases, cognitive biases can distort our view of reality, they can’t always be fixed by just gathering more data, and their effects can add up over time. But when the miscalibrated measuring instrument you’re trying to fix is you, debiasing is a unique challenge.
Still, this is an obvious place to start. For if you can’t trust your brain, how can you trust anything else?
Noticing Bias
Imagine meeting someone for the first time, and knowing nothing about them except that they’re shy.
Question: Is it more likely that this person is a librarian, or a salesperson?
Most people answer “librarian.” Which is a mistake: shy salespeople are much more common than shy librarians, because salespeople in general are much more common than librarians—seventy-five times as common, in the United States.¹
This is base rate neglect: grounding one’s judgments in how well sets of characteristics feel like they fit together, and neglecting how common each characteristic is in the population at large.² Another example of a cognitive bias is the sunk cost fallacy—people’s tendency to feel committed to things they’ve spent resources on in the past, when they should be cutting their losses and moving on.
Knowing about these biases, unfortunately, doesn’t make you immune to them. It doesn’t even mean you’ll be able to notice them in action.
In a study of bias blindness, experimental subjects predicted that they would have a harder time neutrally evaluating the quality of paintings if they knew the paintings were by famous artists. And indeed, these subjects exhibited the very bias they had predicted when the experimenters later tested their prediction. When asked afterward, however, the very same subjects claimed that their assessments of the paintings had been objective and unaffected by the bias.³
Even when we correctly identify others’ biases, we exhibit a bias blind spot when it comes to our own flaws.⁴ Failing to detect any “biased-feeling thoughts” when we introspect, we draw the conclusion that we must just be less biased than everyone else.⁵
Yet it is possible to recognize and overcome biases. It’s just not trivial. It’s known that subjects can reduce base rate neglect, for example, by thinking of probabilities as frequencies of objects or events.
The approach to debiasing in this book is to communicate a systematic understanding of why good reasoning works, and of how the brain falls short of it. To the extent this volume does its job, its approach can be compared to the one described in Serfas (2010), who notes that “years of financially related work experience” didn’t affect people’s susceptibility to the sunk cost bias, whereas “the number of accounting courses attended” did help.
The goal of this book is to lay the groundwork for creating rationality “expertise.” That means acquiring a deep understanding of the structure of a very general problem: human bias, self-deception, and the thousand paths by which sophisticated thought can defeat itself.
A Word About This Text
Map and Territory began its life as a series of essays by decision theorist Eliezer Yudkowsky, published between 2006 and 2009 on the economics blog Overcoming Bias and its spin-off community blog Less Wrong. Thematically linked essays were grouped together in “sequences,” and thematically linked sequences were grouped into books. Map and Territory is the first of six such books, with the series as a whole going by the name Rationality: From AI to Zombies.⁷
In style, this series run the gamut from “lively textbook” to “compendium of vignettes” to “riotous manifesto,” and the content is correspondingly varied. The resultant rationality primer is frequently personal and irreverent—drawing, for example, from Yudkowsky’s experiences with his Orthodox Jewish mother (a psychiatrist) and father (a physicist), and from conversations on chat rooms and mailing lists. Readers who are familiar with Yudkowsky from Harry Potter and the Methods of Rationality, his science-oriented take-off of J.K. Rowling’s Harry Potter books, will recognize the same iconoclasm, and many of the same themes.
The philosopher Alfred Korzybski once wrote: “A map is not the territory it represents, but, if correct, it has a similar structure to the territory, which accounts for its usefulness.” And what can be said of maps here, as Korzybski noted, can also be said of beliefs, and assertions, and words.
“The map is not the territory.” This deceptively simple claim is the organizing idea behind this book, and behind the four sequences of essays collected here: Predictably Wrong, which concerns the systematic ways our beliefs fail to map the real world; Fake Beliefs, on what makes a belief a “map” in the first place; Noticing Confusion, on how this world-mapping thing our brains do actually works; and Mysterious Answers, which collides these points together. The book then concludes with “The Simple Truth,” a stand-alone dialogue on the idea of truth itself.
Humans aren’t rational; but, as behavioral economist Dan Ariely notes, we’re predictably irrational. There are patterns to how we screw up. And there are patterns to how we behave when we don’t screw up. Both admit of fuller understanding, and with it, the hope of leaning on that understanding to build a better future for ourselves.
¹ Wayne Weiten, Psychology: Themes and Variations, Briefer Version, Eighth Edition (Cengage Learning, 2010).
² Richards J. Heuer, Psychology of Intelligence Analysis (Center for the Study of Intelligence, Central Intelligence Agency, 1999) .
³ Katherine Hansen et al., “People Claim Objectivity After Knowingly Using Biased Strategies,” Personality and Social Psychology Bulletin 40, no. 6 (2014): 691–699 .
⁴ Emily Pronin, Daniel Y. Lin, and Lee Ross, “The Bias Blind Spot: Perceptions of Bias in Self versus Others,” Personality and Social Psychology Bulletin 28, no. 3 (2002): 369–381 .
⁵ Joyce Ehrlinger, Thomas Gilovich, and Lee Ross, “Peering Into the Bias Blind Spot: People’s Assessments of Bias in Themselves and Others,” Personality and Social Psychology Bulletin 31, no. 5 (2005): 680–692.
⁶ Sebastian Serfas, Cognitive Biases in the Capital Investment Context: Theoretical Considerations and Empirical Experiments on Violations of Normative Rationality (Springer, 2010).
⁷ The first edition of Rationality: From AI to Zombies was released as a single sprawling ebook, before the series was edited and split up into separate volumes. The full book can also be found on http://lesswrong.com/rationality.