Rationality Reading Group: Part G: Against Rationalization

This is part of a semi-monthly reading group on Eliezer Yudkowsky’s ebook, Rationality: From AI to Zombies. For more information about the group, see the announcement post.


Welcome to the Rationality reading group. This fortnight we discuss Part G: Against Rationalization (pp. 293-339). This post summarizes each article of the sequence, linking to the original LessWrong post where available.

G. Against Rationalization

67. Knowing About Biases Can Hurt PeopleLearning common biases won’t help you obtain truth if you only use this knowledge to attack beliefs you don’t like. Discussions about biases need to first do no harm by emphasizing motivated cognition, the sophistication effect, and dysrationalia, although even knowledge of these can backfire.

68. Update Yourself IncrementallyMany people think that you must abandon a belief if you admit any counterevidence. Instead, change your belief by small increments. Acknowledge small pieces of counterevidence by shifting your belief down a little. Supporting evidence will follow if your belief is true. “Won’t you lose debates if you concede any counterarguments?” Rationality is not for winning debates; it is for deciding which side to join.

69. One Argument Against An ArmyIt is tempting to weigh each counterargument by itself against all supporting arguments. No single counterargument can overwhelm all the supporting arguments, so you easily conclude that your theory was right. Indeed, as you win this kind of battle over and over again, you feel ever more confident in your theory. But, in fact, you are just rehearsing already-known evidence in favor of your view.

70. The Bottom LineIf you first write at the bottom of a sheet of paper, “And therefore, the sky is green!”, it does not matter what arguments you write above it afterward; the conclusion is already written, and it is already correct or already wrong.

71. What Evidence Filtered Evidence? - Someone tells you only the evidence that they want you to hear. Are you helpless? Forced to update your beliefs until you reach their position? No, you also have to take into account what they could have told you but didn’t.

72. RationalizationRationality works forward from evidence to conclusions. Rationalization tries in vain to work backward from favourable conclusions to the evidence. But you cannot rationalize what is not already rational. It is as if “lying” were called “truthization”.

73. A Rational ArgumentYou can’t produce a rational argument for something that isn’t rational. First select the rational choice. Then the rational argument is just a list of the same evidence that convinced you.

74. Avoiding Your Belief’s Real Weak PointsWhen people doubt, they instinctively ask only the questions that have easy answers. When you’re doubting one of your most cherished beliefs, close your eyes, empty your mind, grit your teeth, and deliberately think about whatever hurts the most.

75. Motivated Stopping and Motivated ContinuationWhen the evidence we’ve seen points towards a conclusion that we like or dislike, there is a temptation to stop the search for evidence prematurely, or to insist that more evidence is needed.

76. Fake JustificationWe should be suspicious of our tendency to justify our decisions with arguments that did not actually factor into making said decisions. Whatever process you actually use to make your decisions is what determines your effectiveness as a rationalist.

77. Is That Your True Rejection? - People’s stated reason for a rejection may not be the same as the actual reason for that rejection.

78. Entangled Truths, Contagious LiesGiven that things are often entangled with other things in ways which we do not know, it is really really difficult to tell a perfect lie.

79. Of Lies and Black Swan BlowupsAn extreme example of what happens when you get caught in a web of your own lies.

80. Dark Side EpistemologyIf you want to tell a truly convincing lie, to someone who knows what they’re talking about, you either have to lie about lots of specific object level facts, or about more general laws, or about the laws of thought. Lots of the memes out there about how you learn things originally came from people who were trying to convince other people to believe false statements.


This has been a collection of notes on the assigned sequence for this fortnight. The most important part of the reading group though is discussion, which is in the comments section. Please remember that this group contains a variety of levels of expertise: if a line of discussion seems too basic or too incomprehensible, look around for one that suits you better!

The next reading will cover Part H: Against Doublethink (pp. 343-361). The discussion will go live on Wednesday, 26 August 2015, right here on the discussion forum of LessWrong.