# Pit­falls of Rationality

TagLast edit: 13 Nov 2021 14:31 UTC by

Pitfalls of Rationality are frequent error modes, obstacles or problems that arise when people try to practice rationality, or engage with rationality-related materials. Related concepts include the “valley of bad rationality”.

There are two threads touched in posts under this tag:

1. Things that go wrong when people try to be more rational and they unintentionally end up making things worse.

2. Arguably, why haven’t rationalists visible succeeded at their bold and ambitious goals yet?

Regarding the first point, from Incremental Progress and the Valley:

Ah. Well, here’s the the thing: An incremental step in the direction of rationality, if the result is still irrational in other ways, does not have to yield incrementally more winning.

The optimality theorems that we have for probability theory and decision theory, are for perfect probability theory and decision theory. There is no companion theorem which says that, starting from some flawed initial form, every incremental modification of the algorithm that takes the structure closer to the ideal, must yield an incremental improvement in performance. This has not yet been proven, because it is not, in fact, true.

# Know­ing About Bi­ases Can Hurt People

4 Apr 2007 18:01 UTC
204 points

16 Jan 2020 16:15 UTC
259 points

# Mak­ing your ex­plicit rea­son­ing trust­wor­thy

29 Oct 2010 0:00 UTC
120 points

# Rea­son as memetic im­mune disorder

19 Sep 2009 21:05 UTC
491 points

# Self-Im­prove­ment or Shiny Dis­trac­tion: Why Less Wrong is anti-In­stru­men­tal Rationality

14 Sep 2010 16:17 UTC
154 points

# Ex­treme Ra­tion­al­ity: It’s Not That Great

9 Apr 2009 2:44 UTC
242 points

# Mis­takes with Con­ser­va­tion of Ex­pected Evidence

8 Jun 2019 23:07 UTC
222 points

3 Aug 2012 17:52 UTC
69 points

# My ex­pe­rience with the “ra­tio­nal­ist un­canny valley”

23 Apr 2020 20:27 UTC
66 points

# Sum­mary of “The Straw Vul­can”

26 Dec 2011 16:29 UTC
51 points

# The Treach­er­ous Path to Rationality

9 Oct 2020 15:34 UTC
205 points

# In praise of heuristics

24 Oct 2018 15:44 UTC
39 points

24 Mar 2020 21:53 UTC
84 points

# Bucket Errors

29 Jul 2022 18:50 UTC
42 points

# Map Er­rors: The Good, The Bad, and The Territory

27 Jun 2020 5:22 UTC
27 points

# In­cre­men­tal Progress and the Valley

4 Apr 2009 16:42 UTC
94 points

# Don’t Make Your Prob­lems Hide

27 Jun 2020 20:24 UTC
59 points

# Why Don’t Ra­tion­al­ists Win?

5 Sep 2015 0:57 UTC
6 points

# Pseudo-Rationality

6 Feb 2018 8:25 UTC
21 points

# Ac­cu­racy Ver­sus Winning

2 Apr 2009 4:47 UTC
12 points

# The Costs of Rationality

3 Mar 2009 18:13 UTC
37 points

# Don’t Pull a Bro­ken Chain

28 Aug 2019 1:21 UTC
30 points

# An Out­side View on Less Wrong’s Advice

7 Jul 2011 4:46 UTC
84 points

# Bayesi­ans vs. Barbarians

14 Apr 2009 23:45 UTC
103 points

# Ra­tion­al­ists should be­ware rationalism

6 Apr 2009 14:16 UTC
32 points

# No Really, Why Aren’t Ra­tion­al­ists Win­ning?

4 Nov 2018 18:11 UTC
36 points

# Trust in Bayes

29 Jan 2008 23:12 UTC
39 points

# Do we have it too easy?

2 Nov 2011 1:53 UTC
37 points

# I’m be­com­ing in­tol­er­ant. Help.

30 Jun 2011 15:30 UTC
34 points

# Skills and Antiskills

26 Apr 2014 6:54 UTC
43 points

# A Re­fu­ta­tion of (Global) “Hap­piness Max­i­miza­tion”

25 Aug 2020 20:33 UTC
−2 points

# Dig­ging From The Pit­falls of Rationality

19 Jul 2023 4:52 UTC
6 points
(utilitymonster.substack.com)

# Cat­e­go­riza­tion Hell

24 Sep 2023 18:18 UTC
1 point
(utilitymonster.substack.com)

# Do not ask what ra­tio­nal­ists should do

13 Jul 2011 10:47 UTC
34 points

# Two Kinds of Ir­ra­tional­ity and How to Avoid One of Them

2 Feb 2012 18:13 UTC
30 points

# Post/​Pre

6 Mar 2020 0:54 UTC
17 points
(ouroborusdotblog.wordpress.com)

# What can thought-ex­per­i­ments do?

17 Jan 2023 0:35 UTC
16 points

# [Question] What ra­tio­nal­ity failure modes are there?

19 Jan 2024 9:12 UTC
42 points

# Jor­dan Peter­son: Guru/​Villain

3 Feb 2023 9:02 UTC
−14 points

# Why There Is No An­swer to Your Philo­soph­i­cal Question

24 Mar 2023 23:22 UTC
−12 points

# Some po­ten­tial dan­gers of ra­tio­nal­ity training

21 Jan 2012 4:50 UTC
28 points

# The Cow­pox of Doubt

16 Apr 2014 1:04 UTC
57 points

# 18-month fol­low-up on my self-con­cept work

18 Dec 2018 17:40 UTC
62 points
(kajsotala.fi)

# What it means to optimise

25 Jul 2020 9:40 UTC
5 points
(www.neelnanda.io)

# “No ev­i­dence” as a Valley of Bad Rationality

28 Mar 2020 23:45 UTC
106 points

# The prob­lem with too many ra­tio­nal memes

19 Jan 2012 0:56 UTC
121 points

# Dou­ble-Dip­ping in Dun­ning—Kruger

28 Nov 2018 3:40 UTC
88 points

# Against Shoot­ing Your­self in the Foot

16 Nov 2017 20:13 UTC
47 points

# Cul­tish Countercultishness

30 Dec 2007 0:53 UTC
95 points

# Poll re­sults: LW prob­a­bly doesn’t cause akrasia

16 Nov 2011 18:03 UTC
73 points

# Sim­pli­cio and Sophisticus

22 Jul 2018 13:30 UTC
42 points
(thezvi.wordpress.com)

# I’m scared.

23 Dec 2010 9:05 UTC
56 points

# Be­ing Half-Ra­tional About Pas­cal’s Wager is Even Worse

18 Apr 2013 5:20 UTC
63 points

# Goals for which Less Wrong does (and doesn’t) help

18 Nov 2010 22:37 UTC
87 points

# The Sin of Underconfidence

20 Apr 2009 6:30 UTC
99 points

# Defect­ing by Ac­ci­dent—A Flaw Com­mon to An­a­lyt­i­cal People

1 Dec 2010 8:25 UTC
119 points

# Too Smart for My Own Good

22 Jan 2019 17:51 UTC
38 points

# Self-Con­grat­u­la­tory Rationalism

1 Mar 2014 8:52 UTC
73 points

# Fire­wal­ling the Op­ti­mal from the Rational

8 Oct 2012 8:01 UTC
167 points

14 Nov 2018 23:18 UTC
40 points

# Truth: It’s Not That Great

4 May 2014 22:07 UTC
50 points

# Take heed, for it is a trap

14 Aug 2011 10:23 UTC
54 points

# Se­duced by Imagination

16 Jan 2009 3:10 UTC
43 points

# De-Cen­ter­ing Bias

18 Oct 2017 23:24 UTC
14 points