Pit­falls of Rationality

TagLast edit: 13 Nov 2021 14:31 UTC by Yoav Ravid

Pitfalls of Rationality are frequent error modes, obstacles or problems that arise when people try to practice rationality, or engage with rationality-related materials. Related concepts include the “valley of bad rationality”.

There are two threads touched in posts under this tag:

  1. Things that go wrong when people try to be more rational and they unintentionally end up making things worse.

  2. Arguably, why haven’t rationalists visible succeeded at their bold and ambitious goals yet?

Regarding the first point, from Incremental Progress and the Valley:

Ah. Well, here’s the the thing: An incremental step in the direction of rationality, if the result is still irrational in other ways, does not have to yield incrementally more winning.

The optimality theorems that we have for probability theory and decision theory, are for perfect probability theory and decision theory. There is no companion theorem which says that, starting from some flawed initial form, every incremental modification of the algorithm that takes the structure closer to the ideal, must yield an incremental improvement in performance. This has not yet been proven, because it is not, in fact, true.

See also: Criticisms of the Rationalist Movement, Value of Rationality

Know­ing About Bi­ases Can Hurt People

Eliezer Yudkowsky4 Apr 2007 18:01 UTC
178 points
82 comments2 min readLW link

Real­ity-Re­veal­ing and Real­ity-Mask­ing Puzzles

AnnaSalamon16 Jan 2020 16:15 UTC
251 points
61 comments13 min readLW link1 review

Mak­ing your ex­plicit rea­son­ing trust­wor­thy

AnnaSalamon29 Oct 2010 0:00 UTC
119 points
95 comments6 min readLW link

Self-Im­prove­ment or Shiny Dis­trac­tion: Why Less Wrong is anti-In­stru­men­tal Rationality

patrissimo14 Sep 2010 16:17 UTC
140 points
261 comments16 min readLW link

Rea­son as memetic im­mune disorder

PhilGoetz19 Sep 2009 21:05 UTC
417 points
181 comments5 min readLW link

Ex­treme Ra­tion­al­ity: It’s Not That Great

Scott Alexander9 Apr 2009 2:44 UTC
231 points
281 comments8 min readLW link

My ex­pe­rience with the “ra­tio­nal­ist un­canny valley”

Thomas Kwa23 Apr 2020 20:27 UTC
65 points
18 comments5 min readLW link

“Epiphany ad­dic­tion”

cousin_it3 Aug 2012 17:52 UTC
69 points
94 comments1 min readLW link

Mis­takes with Con­ser­va­tion of Ex­pected Evidence

abramdemski8 Jun 2019 23:07 UTC
180 points
22 comments12 min readLW link1 review

Sum­mary of “The Straw Vul­can”

alexvermeer26 Dec 2011 16:29 UTC
49 points
27 comments6 min readLW link

The Treach­er­ous Path to Rationality

Jacob Falkovich9 Oct 2020 15:34 UTC
190 points
116 comments11 min readLW link1 review

No Really, Why Aren’t Ra­tion­al­ists Win­ning?

Sailor Vulcan4 Nov 2018 18:11 UTC
25 points
89 comments5 min readLW link

Why Don’t Ra­tion­al­ists Win?

Adam Zerner5 Sep 2015 0:57 UTC
5 points
118 comments6 min readLW link


Chris_Leong6 Feb 2018 8:25 UTC
19 points
18 comments2 min readLW link

Trust in Bayes

Eliezer Yudkowsky29 Jan 2008 23:12 UTC
36 points
28 comments8 min readLW link

Ad­ding Up To Normality

orthonormal24 Mar 2020 21:53 UTC
81 points
22 comments3 min readLW link

Map Er­rors: The Good, The Bad, and The Territory

orthonormal27 Jun 2020 5:22 UTC
26 points
4 comments2 min readLW link

Don’t Make Your Prob­lems Hide

orthonormal27 Jun 2020 20:24 UTC
57 points
5 comments2 min readLW link

In­cre­men­tal Progress and the Valley

Eliezer Yudkowsky4 Apr 2009 16:42 UTC
85 points
114 comments5 min readLW link

An Out­side View on Less Wrong’s Advice

Mass_Driver7 Jul 2011 4:46 UTC
82 points
162 comments8 min readLW link

Bayesi­ans vs. Barbarians

Eliezer Yudkowsky14 Apr 2009 23:45 UTC
90 points
285 comments8 min readLW link

Ra­tion­al­ists should be­ware rationalism

Kaj_Sotala6 Apr 2009 14:16 UTC
32 points
32 comments3 min readLW link

In praise of heuristics

Bucky24 Oct 2018 15:44 UTC
39 points
27 comments7 min readLW link

Ac­cu­racy Ver­sus Winning

John_Maxwell2 Apr 2009 4:47 UTC
12 points
77 comments1 min readLW link

The Costs of Rationality

RobinHanson3 Mar 2009 18:13 UTC
37 points
81 comments1 min readLW link

Don’t Pull a Bro­ken Chain

johnswentworth28 Aug 2019 1:21 UTC
27 points
6 comments5 min readLW link

Bucket Errors

CFAR!Duncan29 Jul 2022 18:50 UTC
38 points
7 comments11 min readLW link

18-month fol­low-up on my self-con­cept work

Kaj_Sotala18 Dec 2018 17:40 UTC
58 points
4 comments8 min readLW link

What it means to optimise

Neel Nanda25 Jul 2020 9:40 UTC
5 points
0 comments8 min readLW link

“No ev­i­dence” as a Valley of Bad Rationality

Adam Zerner28 Mar 2020 23:45 UTC
104 points
21 comments2 min readLW link1 review

The prob­lem with too many ra­tio­nal memes

Swimmer96319 Jan 2012 0:56 UTC
119 points
342 comments3 min readLW link

Dou­ble-Dip­ping in Dun­ning—Kruger

isovector28 Nov 2018 3:40 UTC
85 points
31 comments3 min readLW link

Against Shoot­ing Your­self in the Foot

Eliezer Yudkowsky16 Nov 2017 20:13 UTC
43 points
3 comments3 min readLW link

Cul­tish Countercultishness

Eliezer Yudkowsky30 Dec 2007 0:53 UTC
79 points
31 comments9 min readLW link

Poll re­sults: LW prob­a­bly doesn’t cause akrasia

AnnaSalamon16 Nov 2011 18:03 UTC
73 points
109 comments8 min readLW link

Sim­pli­cio and Sophisticus

Zvi22 Jul 2018 13:30 UTC
42 points
1 comment4 min readLW link

I’m scared.

Mass_Driver23 Dec 2010 9:05 UTC
56 points
81 comments1 min readLW link

Be­ing Half-Ra­tional About Pas­cal’s Wager is Even Worse

Eliezer Yudkowsky18 Apr 2013 5:20 UTC
58 points
166 comments9 min readLW link

Goals for which Less Wrong does (and doesn’t) help

AnnaSalamon18 Nov 2010 22:37 UTC
86 points
104 comments4 min readLW link

The Sin of Underconfidence

Eliezer Yudkowsky20 Apr 2009 6:30 UTC
93 points
185 comments6 min readLW link

Defect­ing by Ac­ci­dent—A Flaw Com­mon to An­a­lyt­i­cal People

lionhearted1 Dec 2010 8:25 UTC
116 points
432 comments15 min readLW link

Too Smart for My Own Good

isovector22 Jan 2019 17:51 UTC
38 points
4 comments3 min readLW link

Self-Con­grat­u­la­tory Rationalism

ChrisHallquist1 Mar 2014 8:52 UTC
72 points
395 comments10 min readLW link

Fire­wal­ling the Op­ti­mal from the Rational

Eliezer Yudkowsky8 Oct 2012 8:01 UTC
162 points
338 comments5 min readLW link

Sto­icism: Cau­tion­ary Advice

VivaLaPanda14 Nov 2018 23:18 UTC
36 points
16 comments3 min readLW link

Truth: It’s Not That Great

ChrisHallquist4 May 2014 22:07 UTC
50 points
58 comments4 min readLW link

Take heed, for it is a trap

Zed14 Aug 2011 10:23 UTC
56 points
189 comments6 min readLW link

Se­duced by Imagination

Eliezer Yudkowsky16 Jan 2009 3:10 UTC
40 points
20 comments3 min readLW link

De-Cen­ter­ing Bias

Chris_Leong18 Oct 2017 23:24 UTC
14 points
10 comments2 min readLW link

[Link] Nerds are nuts

[deleted]7 Jun 2012 7:48 UTC
35 points
44 comments8 min readLW link

Memes and Ra­tional Decisions

inferential9 Jan 2015 6:42 UTC
35 points
18 comments10 min readLW link

Do we have it too easy?

Klao2 Nov 2011 1:53 UTC
37 points
44 comments3 min readLW link

I’m be­com­ing in­tol­er­ant. Help.

loup-vaillant30 Jun 2011 15:30 UTC
33 points
87 comments1 min readLW link

Skills and Antiskills

katydee26 Apr 2014 6:54 UTC
43 points
54 comments1 min readLW link

A Re­fu­ta­tion of (Global) “Hap­piness Max­i­miza­tion”

fare25 Aug 2020 20:33 UTC
−2 points
4 comments15 min readLW link

Do not ask what ra­tio­nal­ists should do

thakil13 Jul 2011 10:47 UTC
34 points
21 comments1 min readLW link

Two Kinds of Ir­ra­tional­ity and How to Avoid One of Them

ChrisHallquist2 Feb 2012 18:13 UTC
30 points
9 comments4 min readLW link


Ouroborus6 Mar 2020 0:54 UTC
14 points
1 comment1 min readLW link

Some po­ten­tial dan­gers of ra­tio­nal­ity training

lukeprog21 Jan 2012 4:50 UTC
28 points
48 comments1 min readLW link

The Cow­pox of Doubt

Scott Alexander16 Apr 2014 1:04 UTC
48 points
1 comment5 min readLW link