SotW: Avoid Motivated Cognition

(The Ex­er­cise Prize se­ries of posts is the Cen­ter for Ap­plied Ra­tion­al­ity ask­ing for help in­vent­ing ex­er­cises that can teach cog­ni­tive skills. The difficulty is com­ing up with ex­er­cises in­ter­est­ing enough, with a high enough he­do­nic re­turn, that peo­ple ac­tu­ally do them and re­mem­ber them; this of­ten in­volves stand­ing up and perform­ing ac­tions, or in­ter­act­ing with other peo­ple, not just work­ing alone with an ex­er­cise book­let and a pen­cil. We offer prizes of $50 for any sug­ges­tion we de­cide to test, and $500 for any sug­ges­tion we de­cide to adopt. This prize also ex­tends to LW meetup ac­tivi­ties and good ideas for ver­ify­ing that a skill has been ac­quired. See here for de­tails.)


The fol­low­ing awards have been made: $550 to Pal­la­dias, $550 to Stefie_K, $50 to lin­coln­quirk, and $50 to John_Maxwell_IV. See the bot­tom for de­tails. If you’ve earned a prize, please PM StephenCole to claim it. (If you strongly be­lieve that one of your sug­ges­tions Really Would Have Worked, con­sider try­ing it at your lo­cal Less Wrong meetup. If it works there, send us some par­ti­ci­pant com­ments; this may make us up­date enough to test it.)


Lucy and Marvin are walk­ing down the street one day, when they pass a shop show­ing a large choco­late cake in the win­dow.

“Hm,” says Lucy, “I think I’ll buy and eat that choco­late cake.”

“What, the whole thing?” says Marvin. “Now?”

“Yes,” says Lucy, “I want to sup­port the sugar in­dus­try.”

There is a slight pause.

“I don’t sup­pose that your lik­ing choco­late cake has any­thing to do with your de­ci­sion?” says Marvin.

“Well,” says Lucy, “I sup­pose it could have played a role in sug­gest­ing that I eat a whole choco­late cake, but the rea­son why I de­cided to do it was to sup­port the sugar in­dus­try. Lots of peo­ple have jobs in the sugar in­dus­try, and they’ve been hav­ing some trou­ble lately.”


Mo­ti­vated cog­ni­tion is the way (all? most?) brains gen­er­ate false land­scapes of jus­tifi­ca­tion in the pres­ence of at­tach­ments and flinches. It’s not enough for the hu­man brain to at­tach to the sunk cost of a PhD pro­gram, so that we are im­pel­led in our ac­tions to stay—no, that at­tach­ment can also go off and spin a jus­tifi­ca­tional land­scape to con­vince the other parts of our­selves, even the part that knows about con­se­quen­tial­ism and the sunk cost fal­lacy, to stay in the PhD pro­gram.

We’re al­most cer­tain that the sub­ject mat­ter of “mo­ti­vated cog­ni­tion” isn’t a sin­gle unit, prob­a­bly more like 3 or 8 units. We’re also highly un­cer­tain of where to start teach­ing it. Where we start will prob­a­bly end up be­ing de­ter­mined by where we get the best sug­ges­tions for ex­er­cises that can teach it—i.e., end up be­ing de­ter­mined by what we (the com­mu­nity) can figure out how to teach well.

The cog­ni­tive pat­terns that we use to ac­tu­ally com­bat mo­ti­vated cog­ni­tion seem to break out along the fol­low­ing lines:

  1. Our con­cep­tual un­der­stand­ing of ‘mo­ti­vated cog­ni­tion’, and why it’s defec­tive as a cog­ni­tive al­gorithm—the “Bot­tom Line” in­sight.

  2. Ways to re­duce the strength of the ra­tio­nal­iza­tion im­pulse, or re­store truth-seek­ing in the pres­ence of mo­ti­va­tion: e.g., Anna’s “Be­come Cu­ri­ous” tech­nique.

  3. Notic­ing the in­ter­nal at­tach­ment or in­ter­nal flinch, so that you can in­voke the other skills; re­al­iz­ing when you’re in a situ­a­tion that makes you li­able to ra­tio­nal­ize.

  4. Real­ign­ing the in­ter­nal parts that are try­ing to per­suade each other: be­lief-alief or goal-urge rec­on­cili­a­tion pro­ce­dures.

And also:

  • Pat­tern recog­ni­tion of the many styles of warped jus­tifi­ca­tion land­scape that ra­tio­nal­iza­tion cre­ates—be­ing able to rec­og­nize “mo­ti­vated skep­ti­cism” or “re­hears­ing the ev­i­dence” or “mo­ti­vated un­cer­tainty”.

  • Spe­cific coun­ters to ra­tio­nal­iza­tion styles, like “Set bet­ting odds” as a counter to mo­ti­vated un­cer­tainty.

Ex­er­cises to teach all of these are de­sired, but I’m set­ting apart the Ra­tion­al­iza­tion Pat­terns into a sep­a­rate SotW, since there are so many that I’m wor­ried 1-4 won’t get fair treat­ment oth­er­wise. This SotW will fo­cus on items 1-3 above; #4 seems like more of a sep­a­rate unit.


Con­cep­tual un­der­stand­ing /​ in­sights /​ the­o­ret­i­cal back­ground:

The core rea­sons why ra­tio­nal­iza­tion doesn’t work are given in The Bot­tom Line and Ra­tion­al­iza­tion. The Bayesian anal­y­sis of se­lec­tive search is given in What Ev­i­dence Filtered Ev­i­dence? and Con­ser­va­tion of Ex­pected Ev­i­dence.

For fur­ther dis­cus­sion, see the en­tire Against Ra­tion­al­iza­tion se­quence, also The Med­i­ta­tion on Cu­ri­os­ity (for the Li­tany of Tarski).

Some key con­cepts (it’d be nice if some ex­er­cise taught a gut-level un­der­stand­ing thereof, al­though as always the goal is to t each skills rather than con­cepts):

  • Once you write down the an­swer on the bot­tom line of a piece of pa­per in pen, it’s already right or already wrong, and won’t change re­gard­less of what clever ar­gu­ments you write on the lines above.

  • What de­ter­mines your life out­come isn’t how clev­erly you ar­gue for the fore­gone con­clu­sion—what de­ter­mines life out­comes is the al­gorithm that chooses which side to ar­gue for, what you ac­tu­ally do.

  • Ra­tion­al­ity isn’t some­thing you can use to ar­gue for your side; you can never say “Please come up with a ra­tio­nal ar­gu­ment for X”; the only chance for ra­tio­nal­ity to op­er­ate is when you’re de­cid­ing which side to be on.

  • Ev­i­dence that has passed through a filter takes on a differ­ent Bayesian im­port. (Cur­rently han­dled in the Bayes unit.)

  • It is im­pos­si­ble for a ra­tio­nal agent to search for ev­i­dence to look at that will send their be­liefs in a pre­de­ter­mined di­rec­tion. (Cur­rently han­dled in the Bayes unit.)

  • There’s such a thing as a cor­rect prob­a­bil­ity to as­sign to an un­cer­tain propo­si­tion, and this in turn de­ter­mines the weight to lend that pos­si­bil­ity in our ac­tions. Even when things are un­cer­tain, any cog­ni­tion that makes you put too much ac­tion-weight on the wrong be­lief or choice is screw­ing you up.

  • If you’re se­lec­tive about where you look for flaws, or how hard you look for flaws, ev­ery new fal­lacy you learn how to de­tect makes you that much stupi­der.

(We might also need an ex­er­cise just for get­ting peo­ple to un­der­stand the con­cept of mo­ti­vated cog­ni­tion at all. When Anna and Michael ran their first ses­sion on mo­ti­vated cog­ni­tion, they found that while most par­ti­ci­pants im­me­di­ately rec­og­nized the no­tion of ‘ra­tio­nal­iza­tion’ from ex­am­ples like Lucy above, sev­eral peo­ple had no idea what they were talk­ing about—they didn’t see why any­one would ever want to use a tech­nique like the Li­tany of Tarski. Yes, we know you’re skep­ti­cal, we also couldn’t see how that could pos­si­bly be true a pri­ori, but some­times the ev­i­dence just punches you in the nose. After some in­ves­ti­ga­tion, it seems en­tirely pos­si­ble that Ali­corn has sim­ply never ra­tio­nal­ized, ever. Other cases (not Ali­corn’s) sug­gest that some peo­ple might have a very low need for ver­bal jus­tifi­ca­tion; even if they feel guilty about break­ing their diet, they feel no urge to in­vent an elab­o­rate ex­cuse—they just break their diet. On the other hand, LW!Hermione failed to re­pro­duce this ex­per­i­ment—she couldn’t find any­one who didn’t im­me­di­ately rec­og­nize “ra­tio­nal­iza­tion” af­ter 10 tries with her friends. We no­tice we are con­fused.)

(The up­shot is that part of the challenge of con­struct­ing a first unit on mo­ti­vated cog­ni­tion may be to “Ex­plain to some par­ti­ci­pants what the heck a ‘ra­tio­nal­iza­tion’ is, when they don’t re­mem­ber any in­ter­nal ex­pe­rience of that” or might even be “Filter out at­ten­dees who don’t ra­tio­nal­ize in the first place, and have them do a differ­ent unit in­stead.” Please don’t be fas­ci­nated by this prob­lem at the ex­pense of the pri­mary pur­pose of the unit, though; we’re prob­a­bly go­ing to award at most 1 prize on this subtopic, and more likely 0, and there’s an ex­ist­ing thread for fur­ther dis­cus­sion.)


Coun­ter­ing the ra­tio­nal­iza­tion im­pulse /​ restor­ing truth-seek­ing:

The Tarski method: This is the new name of what we were pre­vi­ously call­ing the Li­tany of Tarski: “If the sky is blue, I want to be­lieve the sky is blue; if the sky is not blue, I want to be­lieve the sky is not blue; let me not be­come at­tached to be­liefs I may not want.”

Ex­am­ple: Sup­pose you walk out­side on a fall day wear­ing a short-sleeved shirt, when you feel a slightly chill breath of air on your arms. You won­der if you should go back into the house and get a sweater. But that seems like work; and so your mind quickly notes that the Sun might come out soon and then you wouldn’t need the sweater.

Di­a­gram:

<col><col><col>
It stays cold enough to re­quire a sweater It gets warm enough that no sweater is needed.
You be­lieve you need a sweater A warm walk in a toasty sweater. Your walk is ru­ined for­ever by the need to carry an ex­tra sweater.
You be­lieve you don’t need a sweater You are cold! Cold cold cold! Why didn’t you get a sweater? Free and un­en­cum­bered, you stroll along as the warm Sun comes out over­head.

Vi­su­al­iz­ing all 4 quad­rants of this bi­nary propo­si­tion—the world is like A and I be­lieve A, the world is like B and I be­lieve A, etc. - should, in prin­ci­ple, emo­tion­ally con­firm the truth of the propo­si­tion: “If it will be cold, I want to be­lieve it’s cold; if it’s not cold, I want to be­lieve it’s not cold; let me not be­come at­tached to be­liefs I may not want.”

Eliezer and Anna, when us­ing this method against the temp­ta­tion to be­lieve X, vi­su­al­ize only the quad­rant “The world is not like X and I be­lieve X” to re­mind them­selves of the con­se­quences; e.g. we would only vi­su­al­ize the “You are cold!” quad­rant. Michael Smith (aka “Val”, short for Valen­tine) says that af­ter some prac­tice on this tech­nique as a kata, he was able to vi­su­al­ize all 4 quad­rants quickly and that vi­su­al­iz­ing all 4 seemed to help.

Val also used an up­side-down W-di­a­gram with the two wor­lds at the top and the four be­liefs at the bot­tom, to em­pha­size the idea that the world is there first, and is fixed, and we have only a choice of what to be­lieve within a fixed world, not a choice of which back­ground world to live in. The Tarski Method em­bod­ies a “Start from the world” men­tal pro­cess in which you vi­su­al­ize the world be­ing there first, and your be­lief com­ing af­ter­ward; a similar “Start from the world” rule is like­wise em­pha­sized in the Bayes unit, wherein one starts from a world and asks about the prob­a­bil­ity of the ev­i­dence, rather than start­ing from the ev­i­dence and try­ing to make it match up with a world.

When we ac­tu­ally tested a unit based on ask­ing peo­ple to draw Tarski squares, it didn’t work very well—pos­si­bly be­cause peo­ple didn’t seem to un­der­stand what it was for, or when they would use it; pos­si­bly be­cause it wasn’t a group ex­er­cise. In any case, we already tried teach­ing this the ob­vi­ous way (“Go draw Tarski squares!”) and it didn’t work. But it still seems worth teach­ing if some­one can in­vent a bet­ter ex­er­cise, be­cause it’s some­thing that mul­ti­ple CfAR peo­ple ac­tu­ally use to counter the ra­tio­nal­iza­tion im­pulse /​ re­store truth­seek­ing in real life.

Be­come Cu­ri­ous: De­tect non-cu­ri­os­ity and be­come cu­ri­ous. Anna’s main alarm sig­nal is when she no­tices that she’s not cu­ri­ous in the mid­dle of a con­ver­sa­tion—that she doesn’t have an im­pulse-to-find-out the an­swer—and then try to make her­self cu­ri­ous about the sub­ject of dis­cus­sion. Be­sides vi­su­al­iz­ing the not-X-and-be­lieve-X quad­rant of the Tarski di­a­gram, this is also some­thing you may be able to do by brute in­tro­spec­tion—re­mem­ber the feel­ing of cu­ri­os­ity, and try to call it up. (This is prob­a­bly in the top 3 most im­por­tant things I learned from Anna. -- EY)

Take Pride in Your Ob­jec­tivity: Ju­lia teaches this as a pri­mary counter in her Com­bat Reflexes unit (how to avoid in­stantly defend­ing or at­tack­ing). Eliezer does this ev­ery time he ad­mits he’s wrong on the In­ter­net—con­grat­u­lates him­self on be­ing such a great ra­tio­nal­ist, in or­der to ap­ply counter-he­dons to the flash of pain that would oth­er­wise be as­so­ci­ated.

Vi­su­al­ize a Fixed Prob­a­bil­ity: This is what Eliezer used as a child to stop be­ing scared of the dark—he would de­liber­ately vi­su­al­ize a mur­derer stand­ing with a knife be­hind a door, then vi­su­al­ize his own thoughts hav­ing no effect on the fixed prob­a­bil­ity that any such mur­derer was ac­tu­ally pre­sent. In other words, the no­tion of a “true prob­a­bil­ity” that his thoughts couldn’t af­fect, coun­tered the fear of thoughts af­fect­ing re­al­ity. Vi­su­al­iz­ing there be­ing a fixed fre­quency of wor­lds, or a lawful prob­a­bil­ity that a Bayesian agent would as­sign, can help in per­ceiv­ing the fu­til­ity of ra­tio­nal­iza­tion be­cause you’re try­ing to use ar­gu­ments to move a lawful prob­a­bil­ity that is fixed. This is also part of the do­main of Lawful Uncer­tainty, the no­tion that there are still rules which ap­ply even when we’re un­sure (not presently part of any unit).

Imag­ine the Reve­la­tion: Anna imag­ines that the an­swer is about to be looked up on the In­ter­net, that Omega is about to re­veal the an­swer, etc., to check if her thoughts would change if she was po­ten­tially about to be em­bar­rassed right now. This de­tects be­lief-alief di­ver­gence, but also pro­vides truth­seek­ing im­pulse.

Know­ing the Rules: And fi­nally, if you have suffi­cient mas­tery of prob­a­bil­ity the­ory or de­ci­sion the­ory, you may have a pro­ce­dure to fol­low which is lawful enough, and suffi­ciently well-un­der­stood, that ra­tio­nal­iza­tion can’t in­fluence it much with­out the mis­take be­ing blatant even to you. (In a sense, this is what most of Less Wrong is about—re­duc­ing the amount of self-hon­esty re­quired by in­creas­ing the ob­vi­ous­ness of mis­takes.)


Notic­ing flinches and at­tach­ments, and rais­ing them to con­scious at­ten­tion:

A trig­ger for use of cu­ri­os­ity-restora­tion or the Tarski Method: Notic­ing what it feels like for your mind to:

  • Quickly glimpse a dis­liked ar­gu­ment be­fore sweep­ing it un­der a men­tal rug (flinch)

  • Glimpse a con­clu­sion, find it un­ac­cept­able, quickly start gen­er­at­ing ar­gu­ments against it (flinch)

  • Be cen­tered on a con­clu­sion, au­to­mat­i­cally counter all ar­gu­ments against it (at­tach­ment)

  • In­stantly at­tack a new idea, in­stantly defend an old idea (this is the sub­ject of Ju­lia’s Com­bat Reflexes unit)

Learn­ing to no­tice these events in­tro­spec­tively seems ex­tremely im­por­tant—we all use it heav­ily in daily prac­tice—but we don’t know how to teach that.
Anna ob­serves that Re­jec­tion Ther­apy is of­ten a good time to ob­serve one­self ra­tio­nal­iz­ing, as ap­par­ently many par­ti­ci­pants re­ported that their mind started gen­er­at­ing crazy rea­sons not to ap­proach some­one with a re­quest.
Anna also says that she’s been self-re­ward­ing each time she no­tices a flinch or at­tach­ment, i.e., she’s try­ing to train her in­ner pi­geon to no­tice (not, one hopes, train­ing the flinch­ing or at­tach­ment!) It’s pos­si­ble we could ask par­ti­ci­pants to self-re­ward each event of “notic­ing the flinch or at­tach­ment” while do­ing Re­jec­tion Ther­apy, but we still need other ideas.
Along similar lines of in­ter­nal be­hav­iorism, Eliezer avoids re­ward­ing him­self for ra­tio­nal­iz­ing by re­peat­ing the phrase “Only con­grat­u­late your­self for ac­tu­ally chang­ing a prob­a­bil­ity es­ti­mate or policy” on any oc­ca­sion where he hasn’t changed his mind af­ter ar­gu­ment—as op­posed to e.g. feel­ing any sense of re­ward for hav­ing defeated an in­com­ing ar­gu­ment; even if the in­com­ing ar­gu­ment hap­pens to be wrong, still, “Only con­grat­u­late your­self for ac­tu­ally chang­ing a prob­a­bil­ity es­ti­mate or policy.”
Another thing most of us do is name at­tach­ments or flinches out loud, in con­ver­sa­tion, as we no­tice them, in or­der to re­duce their strength, i.e. “This is prob­a­bly a com­plete post-facto ra­tio­nal­iza­tion, but...” (Eliezer) or “I may just be try­ing to avoid hav­ing my sta­tus re­duced, but...” (Anna). (Note: This re­quires enough trust that nearby peo­ple also know they’re flawed them­selves, that you don’t feel em­bar­rassed for con­fess­ing your own flaws in front of them. In other words, you have to tell em­bar­rass­ing sto­ries about your own failures of ra­tio­nal­ity, be­fore other peo­ple will feel that they can do this around you.)

Anna’s anti-ra­tio­nal­iza­tion makes heavy use of notic­ing sus­pect situ­a­tions where the out­side view says she might ra­tio­nal­ize—cases where her sta­tus is at stake, and so on—and spe­cific key­words like “I be­lieve that” or “No, I re­ally be­lieve that”. She wants to try train­ing peo­ple to no­tice likely con­texts for ra­tio­nal­iza­tion, and to figure out key­words that might in­di­cate ra­tio­nal­iza­tion in them­selves. (Eliezer has never tried to train him­self to no­tice key­words be­cause he figures his brain will just train it­self to avoid the trig­ger phrase; and he wor­ries about likely-con­text train­ing be­cause he’s seen failure modes where no amount of ev­i­dence or sound ar­gu­ment is enough to over­come the sus­pi­cion of ra­tio­nal­iza­tion once it’s been in­voked.)

“Look to­ward the painful thought in­stead of away from it” is an im­por­tant re­flex to in­stall to counter flinches, but would prob­a­bly re­quire some sort of he­do­nic sup­port—like a strong, pre-ex­ist­ing pride in ob­jec­tivity, or a so­cial sup­port group that ap­plauds, or some­thing to stop this from be­ing pure nega­tive re­in­force­ment.

Awards for pre­vi­ous SotW sug­ges­tions:

$550 to Pal­la­dias for the Mon­day-Tues­day game, which has been tested ($50) and now adopted ($500) into the Be Spe­cific unit (though it might be moved to some sort of An­ti­ci­pa­tion unit later on).

$550 to Stefie_K for her sug­ges­tion to have the in­struc­tor pre­tend to be some­one who re­ally wants you to in­vest in their com­pany, but is never spe­cific; also $50 to daen­rys for the “More Spe­cific!” im­prov-game sug­ges­tion. In com­bi­na­tion these in­spired the Vague Con­sul­tant game (“Hi, I’m a con­sul­tant, I’m here to im­prove your busi­ness pro­cesses!” “How?” “By con­sult­ing with stake­hold­ers!”) which has now been adopted into the Be Spe­cific unit.

$50 to lin­coln­quirk for the “Chan­nel Paul Gra­ham” game, which we tested. We all thought this would work—it was our high­est-rated can­di­date sug­ges­tion—but it didn’t get pos­i­tive au­di­ence feed­back. Con­grat­u­la­tions to lin­coln­quirk on a good sug­ges­tion nonethe­less.

We haven’t yet tested, but definitely in­tend to at least test, and are hence already award­ing $50 to, the fol­low­ing idea:

$50 to John Maxwell IV for the Choose Your Own Ad­ven­ture sug­ges­tion for the Con­se­quen­tial­ism unit.

To claim a prize, send a LessWrong pri­vate mes­sage (so we know it origi­nates from the same LW user ac­count) to StephenCole.