Pit­falls of Rationality

TagLast edit: 13 Nov 2021 14:31 UTC by Yoav Ravid

Pitfalls of Rationality are frequent error modes, obstacles or problems that arise when people try to practice rationality, or engage with rationality-related materials. Related concepts include the “valley of bad rationality”.

There are two threads touched in posts under this tag:

  1. Things that go wrong when people try to be more rational and they unintentionally end up making things worse.

  2. Arguably, why haven’t rationalists visible succeeded at their bold and ambitious goals yet?

Regarding the first point, from Incremental Progress and the Valley:

Ah. Well, here’s the the thing: An incremental step in the direction of rationality, if the result is still irrational in other ways, does not have to yield incrementally more winning.

The optimality theorems that we have for probability theory and decision theory, are for perfect probability theory and decision theory. There is no companion theorem which says that, starting from some flawed initial form, every incremental modification of the algorithm that takes the structure closer to the ideal, must yield an incremental improvement in performance. This has not yet been proven, because it is not, in fact, true.

See also: Criticisms of the Rationalist Movement, Value of Rationality

Know­ing About Bi­ases Can Hurt People

Eliezer Yudkowsky4 Apr 2007 18:01 UTC
188 points
82 comments2 min readLW link

Real­ity-Re­veal­ing and Real­ity-Mask­ing Puzzles

AnnaSalamon16 Jan 2020 16:15 UTC
252 points
57 comments13 min readLW link1 review

Mak­ing your ex­plicit rea­son­ing trust­wor­thy

AnnaSalamon29 Oct 2010 0:00 UTC
119 points
95 comments6 min readLW link

Self-Im­prove­ment or Shiny Dis­trac­tion: Why Less Wrong is anti-In­stru­men­tal Rationality

patrissimo14 Sep 2010 16:17 UTC
149 points
261 comments16 min readLW link

Rea­son as memetic im­mune disorder

PhilGoetz19 Sep 2009 21:05 UTC
439 points
183 comments5 min readLW link

Ex­treme Ra­tion­al­ity: It’s Not That Great

Scott Alexander9 Apr 2009 2:44 UTC
240 points
281 comments8 min readLW link

Mis­takes with Con­ser­va­tion of Ex­pected Evidence

abramdemski8 Jun 2019 23:07 UTC
196 points
24 comments12 min readLW link1 review

“Epiphany ad­dic­tion”

cousin_it3 Aug 2012 17:52 UTC
69 points
90 comments1 min readLW link

My ex­pe­rience with the “ra­tio­nal­ist un­canny valley”

Thomas Kwa23 Apr 2020 20:27 UTC
66 points
18 comments5 min readLW link

The Treach­er­ous Path to Rationality

Jacob Falkovich9 Oct 2020 15:34 UTC
196 points
115 comments11 min readLW link1 review

Sum­mary of “The Straw Vul­can”

alexvermeer26 Dec 2011 16:29 UTC
49 points
27 comments6 min readLW link

Map Er­rors: The Good, The Bad, and The Territory

orthonormal27 Jun 2020 5:22 UTC
27 points
4 comments2 min readLW link

Don’t Make Your Prob­lems Hide

orthonormal27 Jun 2020 20:24 UTC
59 points
5 comments2 min readLW link

Bucket Errors

CFAR!Duncan29 Jul 2022 18:50 UTC
40 points
7 comments11 min readLW link

In­cre­men­tal Progress and the Valley

Eliezer Yudkowsky4 Apr 2009 16:42 UTC
87 points
113 comments5 min readLW link


Chris_Leong6 Feb 2018 8:25 UTC
19 points
18 comments2 min readLW link

No Really, Why Aren’t Ra­tion­al­ists Win­ning?

Sailor Vulcan4 Nov 2018 18:11 UTC
30 points
89 comments5 min readLW link

Ac­cu­racy Ver­sus Winning

John_Maxwell2 Apr 2009 4:47 UTC
12 points
77 comments1 min readLW link

The Costs of Rationality

RobinHanson3 Mar 2009 18:13 UTC
37 points
81 comments1 min readLW link

Don’t Pull a Bro­ken Chain

johnswentworth28 Aug 2019 1:21 UTC
27 points
6 comments5 min readLW link

An Out­side View on Less Wrong’s Advice

Mass_Driver7 Jul 2011 4:46 UTC
84 points
162 comments8 min readLW link

Bayesi­ans vs. Barbarians

Eliezer Yudkowsky14 Apr 2009 23:45 UTC
96 points
277 comments8 min readLW link

Ra­tion­al­ists should be­ware rationalism

Kaj_Sotala6 Apr 2009 14:16 UTC
32 points
32 comments3 min readLW link

Why Don’t Ra­tion­al­ists Win?

Adam Zerner5 Sep 2015 0:57 UTC
6 points
118 comments6 min readLW link

Trust in Bayes

Eliezer Yudkowsky29 Jan 2008 23:12 UTC
37 points
28 comments8 min readLW link

Ad­ding Up To Normality

orthonormal24 Mar 2020 21:53 UTC
82 points
22 comments3 min readLW link

In praise of heuristics

Bucky24 Oct 2018 15:44 UTC
39 points
27 comments7 min readLW link

A Re­fu­ta­tion of (Global) “Hap­piness Max­i­miza­tion”

fare25 Aug 2020 20:33 UTC
−2 points
4 comments15 min readLW link

Do not ask what ra­tio­nal­ists should do

thakil13 Jul 2011 10:47 UTC
34 points
21 comments1 min readLW link

Two Kinds of Ir­ra­tional­ity and How to Avoid One of Them

ChrisHallquist2 Feb 2012 18:13 UTC
30 points
9 comments4 min readLW link


Ouroborus6 Mar 2020 0:54 UTC
14 points
1 comment1 min readLW link

Some po­ten­tial dan­gers of ra­tio­nal­ity training

lukeprog21 Jan 2012 4:50 UTC
28 points
48 comments1 min readLW link

The Cow­pox of Doubt

Scott Alexander16 Apr 2014 1:04 UTC
50 points
1 comment5 min readLW link

What can thought-ex­per­i­ments do?

Cleo Nardo17 Jan 2023 0:35 UTC
16 points
3 comments5 min readLW link

Jor­dan Peter­son: Guru/​Villain

Bryan Frances3 Feb 2023 9:02 UTC
−16 points
6 comments9 min readLW link

Why There Is No An­swer to Your Philo­soph­i­cal Question

Bryan Frances24 Mar 2023 23:22 UTC
−12 points
10 comments12 min readLW link

18-month fol­low-up on my self-con­cept work

Kaj_Sotala18 Dec 2018 17:40 UTC
60 points
4 comments8 min readLW link

What it means to optimise

Neel Nanda25 Jul 2020 9:40 UTC
5 points
0 comments8 min readLW link

“No ev­i­dence” as a Valley of Bad Rationality

Adam Zerner28 Mar 2020 23:45 UTC
104 points
21 comments2 min readLW link1 review

The prob­lem with too many ra­tio­nal memes

Swimmer963 (Miranda Dixon-Luinenburg) 19 Jan 2012 0:56 UTC
121 points
342 comments3 min readLW link

Dou­ble-Dip­ping in Dun­ning—Kruger

isovector28 Nov 2018 3:40 UTC
86 points
31 comments3 min readLW link

Against Shoot­ing Your­self in the Foot

Eliezer Yudkowsky16 Nov 2017 20:13 UTC
45 points
3 comments3 min readLW link

Cul­tish Countercultishness

Eliezer Yudkowsky30 Dec 2007 0:53 UTC
84 points
31 comments9 min readLW link

Poll re­sults: LW prob­a­bly doesn’t cause akrasia

AnnaSalamon16 Nov 2011 18:03 UTC
73 points
106 comments8 min readLW link

Sim­pli­cio and Sophisticus

Zvi22 Jul 2018 13:30 UTC
42 points
1 comment4 min readLW link

I’m scared.

Mass_Driver23 Dec 2010 9:05 UTC
56 points
81 comments1 min readLW link

Be­ing Half-Ra­tional About Pas­cal’s Wager is Even Worse

Eliezer Yudkowsky18 Apr 2013 5:20 UTC
60 points
162 comments9 min readLW link

Goals for which Less Wrong does (and doesn’t) help

AnnaSalamon18 Nov 2010 22:37 UTC
87 points
104 comments4 min readLW link

The Sin of Underconfidence

Eliezer Yudkowsky20 Apr 2009 6:30 UTC
94 points
185 comments6 min readLW link

Defect­ing by Ac­ci­dent—A Flaw Com­mon to An­a­lyt­i­cal People

lionhearted (Sebastian Marshall)1 Dec 2010 8:25 UTC
116 points
432 comments15 min readLW link

Too Smart for My Own Good

isovector22 Jan 2019 17:51 UTC
38 points
4 comments3 min readLW link

Self-Con­grat­u­la­tory Rationalism

ChrisHallquist1 Mar 2014 8:52 UTC
73 points
395 comments10 min readLW link

Fire­wal­ling the Op­ti­mal from the Rational

Eliezer Yudkowsky8 Oct 2012 8:01 UTC
166 points
313 comments5 min readLW link

Sto­icism: Cau­tion­ary Advice

VivaLaPanda14 Nov 2018 23:18 UTC
38 points
16 comments3 min readLW link

Truth: It’s Not That Great

ChrisHallquist4 May 2014 22:07 UTC
50 points
58 comments4 min readLW link

Take heed, for it is a trap

Zed14 Aug 2011 10:23 UTC
56 points
189 comments6 min readLW link

Se­duced by Imagination

Eliezer Yudkowsky16 Jan 2009 3:10 UTC
40 points
20 comments3 min readLW link

De-Cen­ter­ing Bias

Chris_Leong18 Oct 2017 23:24 UTC
14 points
10 comments2 min readLW link

[Link] Nerds are nuts

[deleted]7 Jun 2012 7:48 UTC
35 points
44 comments8 min readLW link

Memes and Ra­tional Decisions

inferential9 Jan 2015 6:42 UTC
35 points
18 comments10 min readLW link

Do we have it too easy?

Klao2 Nov 2011 1:53 UTC
37 points
44 comments3 min readLW link

I’m be­com­ing in­tol­er­ant. Help.

loup-vaillant30 Jun 2011 15:30 UTC
33 points
85 comments1 min readLW link

Skills and Antiskills

katydee26 Apr 2014 6:54 UTC
43 points
54 comments1 min readLW link
No comments.