Pit­falls of Rationality

TagLast edit: 6 Jan 2021 20:39 UTC by Multicore

Pitfalls of Rationality are frequent error modes, obstacles or problems that arise when people try to practice rationality, or engage with rationality-related materials. Related concepts include the “valley of bad rationality”.

There are two threads touched in posts under this tag:

  1. Things that go wrong when people try to be more rational and they unintentionally end up making things worse.

  2. Arguably, why haven’t rationalists visible succeeded at their bold and ambitious goals yet?

Regarding the first point, from Incremental Progress and the Valley:

Ah. Well, here’s the the thing: An incremental step in the direction of rationality, if the result is still irrational in other ways, does not have to yield incrementally more winning.

The optimality theorems that we have for probability theory and decision theory, are for perfect probability theory and decision theory. There is no companion theorem which says that, starting from some flawed initial form, every incremental modification of the algorithm that takes the structure closer to the ideal, must yield an incremental improvement in performance. This has not yet been proven, because it is not, in fact, true.

See also: Criticisms of the Rationalist Movement, Is Rationality Any Good?

Know­ing About Bi­ases Can Hurt People

Eliezer Yudkowsky4 Apr 2007 18:01 UTC
135 points
80 comments2 min readLW link

Real­ity-Re­veal­ing and Real­ity-Mask­ing Puzzles

AnnaSalamon16 Jan 2020 16:15 UTC
238 points
57 comments13 min readLW link

Mak­ing your ex­plicit rea­son­ing trust­wor­thy

AnnaSalamon29 Oct 2010 0:00 UTC
117 points
95 comments6 min readLW link

Self-Im­prove­ment or Shiny Dis­trac­tion: Why Less Wrong is anti-In­stru­men­tal Rationality

patrissimo14 Sep 2010 16:17 UTC
126 points
261 comments16 min readLW link

Ex­treme Ra­tion­al­ity: It’s Not That Great

Scott Alexander9 Apr 2009 2:44 UTC
196 points
277 comments8 min readLW link

My ex­pe­rience with the “ra­tio­nal­ist un­canny valley”

Thomas Kwa23 Apr 2020 20:27 UTC
62 points
18 comments5 min readLW link

“Epiphany ad­dic­tion”

cousin_it3 Aug 2012 17:52 UTC
69 points
94 comments1 min readLW link

Sum­mary of “The Straw Vul­can”

alexvermeer26 Dec 2011 16:29 UTC
49 points
27 comments6 min readLW link

Mis­takes with Con­ser­va­tion of Ex­pected Evidence

abramdemski8 Jun 2019 23:07 UTC
159 points
21 comments12 min readLW link2 nominations1 review

Rea­son as memetic im­mune disorder

PhilGoetz19 Sep 2009 21:05 UTC
340 points
177 comments5 min readLW link

The Treach­er­ous Path to Rationality

Jacob Falkovich9 Oct 2020 15:34 UTC
173 points
114 comments11 min readLW link

No Really, Why Aren’t Ra­tion­al­ists Win­ning?

Sailor Vulcan4 Nov 2018 18:11 UTC
25 points
85 comments5 min readLW link

Why Don’t Ra­tion­al­ists Win?

adamzerner5 Sep 2015 0:57 UTC
4 points
118 comments6 min readLW link


Chris_Leong6 Feb 2018 8:25 UTC
19 points
18 comments2 min readLW link

Trust in Bayes

Eliezer Yudkowsky29 Jan 2008 23:12 UTC
33 points
27 comments8 min readLW link

Ad­ding Up To Normality

orthonormal24 Mar 2020 21:53 UTC
73 points
22 comments3 min readLW link

Map Er­rors: The Good, The Bad, and The Territory

orthonormal27 Jun 2020 5:22 UTC
25 points
4 comments2 min readLW link

Don’t Make Your Prob­lems Hide

orthonormal27 Jun 2020 20:24 UTC
55 points
5 comments2 min readLW link

In­cre­men­tal Progress and the Valley

Eliezer Yudkowsky4 Apr 2009 16:42 UTC
67 points
114 comments5 min readLW link

Bayesi­ans vs. Barbarians

Eliezer Yudkowsky14 Apr 2009 23:45 UTC
77 points
284 comments8 min readLW link

Ra­tion­al­ists should be­ware rationalism

Kaj_Sotala6 Apr 2009 14:16 UTC
31 points
32 comments3 min readLW link

In praise of heuristics

Bucky24 Oct 2018 15:44 UTC
39 points
27 comments7 min readLW link

Ac­cu­racy Ver­sus Winning

John_Maxwell2 Apr 2009 4:47 UTC
12 points
77 comments1 min readLW link

The Costs of Rationality

RobinHanson3 Mar 2009 18:13 UTC
34 points
81 comments1 min readLW link

Don’t Pull a Bro­ken Chain

johnswentworth28 Aug 2019 1:21 UTC
25 points
6 comments5 min readLW link

18-month fol­low-up on my self-con­cept work

Kaj_Sotala18 Dec 2018 17:40 UTC
55 points
4 comments8 min readLW link

What it means to optimise

Neel Nanda25 Jul 2020 9:40 UTC
3 points
0 comments8 min readLW link

“No ev­i­dence” as a Valley of Bad Rationality

adamzerner28 Mar 2020 23:45 UTC
99 points
14 comments2 min readLW link

The prob­lem with too many ra­tio­nal memes

Swimmer96319 Jan 2012 0:56 UTC
108 points
342 comments3 min readLW link

Dou­ble-Dip­ping in Dun­ning—Kruger

isovector28 Nov 2018 3:40 UTC
78 points
31 comments3 min readLW link

Against Shoot­ing Your­self in the Foot

Eliezer Yudkowsky16 Nov 2017 20:13 UTC
38 points
3 comments3 min readLW link

An Out­side View on Less Wrong’s Advice

Mass_Driver7 Jul 2011 4:46 UTC
78 points
160 comments8 min readLW link

Cul­tish Countercultishness

Eliezer Yudkowsky30 Dec 2007 0:53 UTC
62 points
30 comments9 min readLW link

Poll re­sults: LW prob­a­bly doesn’t cause akrasia

AnnaSalamon16 Nov 2011 18:03 UTC
69 points
109 comments8 min readLW link

Sim­pli­cio and Sophisticus

Zvi22 Jul 2018 13:30 UTC
42 points
1 comment4 min readLW link

I’m scared.

Mass_Driver23 Dec 2010 9:05 UTC
54 points
81 comments1 min readLW link

Be­ing Half-Ra­tional About Pas­cal’s Wager is Even Worse

Eliezer Yudkowsky18 Apr 2013 5:20 UTC
41 points
166 comments9 min readLW link

Goals for which Less Wrong does (and doesn’t) help

AnnaSalamon18 Nov 2010 22:37 UTC
82 points
104 comments4 min readLW link

The Sin of Underconfidence

Eliezer Yudkowsky20 Apr 2009 6:30 UTC
74 points
185 comments6 min readLW link

Defect­ing by Ac­ci­dent—A Flaw Com­mon to An­a­lyt­i­cal People

lionhearted1 Dec 2010 8:25 UTC
115 points
431 comments15 min readLW link

Too Smart for My Own Good

isovector22 Jan 2019 17:51 UTC
38 points
4 comments3 min readLW link

Self-Con­grat­u­la­tory Rationalism

ChrisHallquist1 Mar 2014 8:52 UTC
70 points
395 comments10 min readLW link

Fire­wal­ling the Op­ti­mal from the Rational

Eliezer Yudkowsky8 Oct 2012 8:01 UTC
141 points
337 comments5 min readLW link

Sto­icism: Cau­tion­ary Advice

VivaLaPanda14 Nov 2018 23:18 UTC
35 points
16 comments3 min readLW link

Truth: It’s Not That Great

ChrisHallquist4 May 2014 22:07 UTC
50 points
58 comments4 min readLW link

Take heed, for it is a trap

Zed14 Aug 2011 10:23 UTC
54 points
189 comments6 min readLW link

Se­duced by Imagination

Eliezer Yudkowsky16 Jan 2009 3:10 UTC
30 points
20 comments3 min readLW link

De-Cen­ter­ing Bias

Chris_Leong18 Oct 2017 23:24 UTC
14 points
10 comments2 min readLW link

[Link] Nerds are nuts

[deleted]7 Jun 2012 7:48 UTC
34 points
44 comments8 min readLW link

Memes and Ra­tional Decisions

inferential9 Jan 2015 6:42 UTC
35 points
17 comments10 min readLW link

Do we have it too easy?

Klao2 Nov 2011 1:53 UTC
37 points
44 comments3 min readLW link

I’m be­com­ing in­tol­er­ant. Help.

loup-vaillant30 Jun 2011 15:30 UTC
33 points
87 comments1 min readLW link

Skills and Antiskills

katydee26 Apr 2014 6:54 UTC
43 points
54 comments1 min readLW link

A Re­fu­ta­tion of (Global) “Hap­piness Max­i­miza­tion”

fare25 Aug 2020 20:33 UTC
−2 points
4 comments15 min readLW link

Do not ask what ra­tio­nal­ists should do

thakil13 Jul 2011 10:47 UTC
34 points
21 comments1 min readLW link

Two Kinds of Ir­ra­tional­ity and How to Avoid One of Them

ChrisHallquist2 Feb 2012 18:13 UTC
30 points
9 comments4 min readLW link


Ouroborus6 Mar 2020 0:54 UTC
14 points
1 comment1 min readLW link

Some po­ten­tial dan­gers of ra­tio­nal­ity training

lukeprog21 Jan 2012 4:50 UTC
28 points
48 comments1 min readLW link

The Cow­pox of Doubt

Scott Alexander16 Apr 2014 1:04 UTC
15 points
0 comments5 min readLW link
No comments.