Making your explicit reasoning trustworthy

Or: “I don’t want to think about that! I might be left with mis­taken be­liefs!”

Re­lated to: Ra­tion­al­ity as memetic im­mune di­s­or­der; In­cre­men­tal progress and the valley; Egan’s Law.

tl;dr: Many of us hes­i­tate to trust ex­plicit rea­son­ing be­cause… we haven’t built the skills that make such rea­son­ing trust­wor­thy. Some sim­ple strate­gies can help.

Most of us are afraid to think fully about cer­tain sub­jects.

Some­times, we avert our eyes for fear of un­pleas­ant con­clu­sions. (“What if it’s my fault? What if I’m not good enough?”)

But other times, oddly enough, we avert our eyes for fear of in­ac­cu­rate con­clu­sions.[1] Peo­ple fear ques­tion­ing their re­li­gion, lest they dis­be­lieve and be­come damned. Peo­ple fear ques­tion­ing their “don’t walk alone at night” safety strat­egy, lest they ven­ture into dan­ger. And I find I hes­i­tate when pon­der­ing Pas­cal’s wa­ger, in­finite ethics, the Si­mu­la­tion ar­gu­ment, and whether I’m a Boltz­mann brain… be­cause I’m afraid of los­ing my bear­ings, and be­liev­ing mis­taken things.

Ostrich The­ory, one might call it. Or I’m Already Right the­ory. The the­ory that we’re more likely to act sen­si­bly if we don’t think fur­ther, than if we do. Some­times Ostrich The­o­ries are un­con­sciously held; one just word­lessly backs away from cer­tain thoughts. Other times full or par­tial Ostrich The­o­ries are put forth ex­plic­itly, as in Phil Goetz’s post, this LW com­ment, dis­cus­sions of Tet­lock’s “foxes vs hedge­hogs” re­search, en­join­ders to use “out­side views”, en­join­ders not to sec­ond-guess ex­pert sys­tems, and cau­tions for Chris­ti­ans against “clever ar­gu­ments”.

Ex­plicit rea­son­ing is of­ten nuts

Ostrich The­o­ries sound im­plau­si­ble: why would not think­ing through an is­sue make our ac­tions bet­ter? And yet ex­am­ples abound of folks whose the­o­ries and the­o­riz­ing (as con­trasted with their habits, word­less in­tu­itions, and unar­tic­u­lated re­sponses to so­cial pres­sures or their own emo­tions) made sig­nifi­cant chunks of their ac­tions worse. Ex­am­ples in­clude, among many oth­ers:

  • Most early Com­mu­nists;

  • Ted Kaczyn­ski (The Un­abomber; an IQ 160 math PhD who wrote an in­ter­est­ing trea­tise about the hu­man im­pacts of tech­nol­ogy, and also mur­dered in­no­cent peo­ple while ac­com­plish­ing noth­ing);

  • Mitchell Heis­man;

  • Folks who go to great lengths to keep kosher;

  • Friends of mine who’ve gone to great lengths to be metic­u­lously de­no­ta­tion­ally hon­est, in­clud­ing re­fus­ing jobs that re­quired a gov­ern­ment loy­alty oath, and re­fus­ing to click on user agree­ments for videogames; and

  • Many who’ve gone to war for the sake of re­li­gion, na­tional iden­tity, or many differ­ent far-mode ideals.

In fact, the ex­am­ples of re­li­gion and war sug­gest that the trou­ble with, say, Kaczyn­ski wasn’t that his be­liefs were un­usu­ally crazy. The trou­ble was that his be­liefs were an or­di­nary amount of crazy, and he was un­usu­ally prone to act­ing on his be­liefs. If the av­er­age per­son started to ac­tu­ally act on their nom­i­nal, ver­bal, ex­plicit be­liefs, they, too, would in many cases look plumb nuts. For ex­am­ple, a Chris­tian might give away all their pos­ses­sions, re­joice at the death of their chil­dren in cir­cum­stances where they seem likely to have gone to heaven, and gen­er­ally treat their chances of Heaven vs Hell as their top pri­or­ity. Some­one else might risk their life-sav­ings bet­ting on an elec­tion out­come or busi­ness about which they were “99% con­fi­dent”.

That is: many peo­ples’ ab­stract rea­son­ing is not up to the task of day to day de­ci­sion-mak­ing. This doesn’t im­pair folks’ ac­tions all that much, be­cause peo­ples’ ab­stract rea­son­ing has lit­tle bear­ing on our ac­tual ac­tions. Mostly we just find our­selves do­ing things (out of habit, emo­tional in­cli­na­tion, or so­cial copy­ing) and make up the rea­sons post-hoc. But when we do try to choose ac­tions from the­ory, the re­sults are far from re­li­ably helpful—and so many folks’ early steps to­ward ra­tio­nal­ity go un­re­warded.

We are left with two linked bar­ri­ers to ra­tio­nal­ity: (1) nutty ab­stract rea­son­ing; and (2) fears of rea­soned nut­ti­ness, and other failures to be­lieve that think­ing things through is ac­tu­ally helpful.[2]

Rea­son­ing can be made less risky

Much of this nut­ti­ness is un­nec­es­sary. There are learn­able skills that can both make our ab­stract rea­son­ing more trust­wor­thy and also make it eas­ier for us to trust it.

Here’s the ba­sic idea:

If you know the limi­ta­tions of a pat­tern of rea­son­ing, learn­ing bet­ter what it says won’t hurt you. It’s like hav­ing a friend who’s of­ten wrong. If you don’t know your friend’s limi­ta­tions, his ad­vice might harm you. But once you do know, you don’t have to gag him; you can listen to what he says, and then take it with a grain of salt.[3]

Rea­son­ing is the meta-tool that lets us figure out what meth­ods of in­fer­ence are trust­wor­thy where. Rea­son lets us look over the track records of our own ex­plicit the­o­riz­ing, out­side ex­perts’ views, our near-mode in­tu­itions, etc. and figure out which is how trust­wor­thy in a given situ­a­tion.

If we learn to use this meta-tool, we can walk into ra­tio­nal­ity with­out fear.

Skills for safer reasoning

1. Rec­og­nize im­plicit knowl­edge.

Rec­og­nize when your habits, or out­side cus­toms, are likely to work bet­ter than your rea­soned-from-scratch best guesses. No­tice how differ­ent groups act and what re­sults they get. Take pains to stay aware of your own an­ti­ci­pa­tions, es­pe­cially in cases where you have ex­plicit ver­bal mod­els that might block your an­ti­ci­pa­tions from view. And, by study­ing track records, get a sense of which pre­dic­tion meth­ods are trust­wor­thy where.

Use track records; don’t as­sume that just be­cause folks’ jus­tifi­ca­tions are in­co­her­ent, the ac­tions they are jus­tify­ing are fool­ish. But also don’t as­sume that tra­di­tion is bet­ter than your mod­els. Be em­piri­cal.

2. Plan for er­rors in your best-guess mod­els.

We tend to be over­con­fi­dent in our own be­liefs, to over­es­ti­mate the prob­a­bil­ity of con­junc­tions (such as multi-part rea­son­ing chains), and to search prefer­en­tially for ev­i­dence that we’re right. Put these facts to­gether, and the­o­ries folks are “al­most cer­tain” of turn out to be wrong pretty of­ten. There­fore:

  • Make pre­dic­tions from as many an­gles as pos­si­ble, to build re­dun­dancy. Use mul­ti­ple the­o­ret­i­cal frame­works, mul­ti­ple datasets, mul­ti­ple ex­perts, mul­ti­ple dis­ci­plines.

  • When some lines of ar­gu­ment point one way and some an­other, don’t give up or take a vote. In­stead, no­tice that you’re con­fused, and (while guard­ing against con­fir­ma­tion bias!) seek fol­low-up in­for­ma­tion.

  • Use your mem­o­ries of past er­ror to bring up hon­est cu­ri­os­ity and fear of er­ror. Then, re­ally search for ev­i­dence that you’re wrong, the same way you’d search if your life were be­ing bet on some­one else’s the­ory.

  • Build safe­guards, al­ter­na­tives, and re­pur­pos­able re­sources into your plans.

3. Be­ware rapid be­lief changes.

Some peo­ple find their be­liefs chang­ing rapidly back and forth, based for ex­am­ple on the par­tic­u­lar lines of ar­gu­ment they’re cur­rently pon­der­ing, or the be­liefs of those they’ve re­cently read or talked to. Such fluc­tu­a­tions are gen­er­ally bad news for both the ac­cu­racy of your be­liefs and the use­ful­ness of your ac­tions. If this is your situ­a­tion:

  • Re­mem­ber that ac­cu­rate be­liefs come from an even, long-term col­lec­tion of all the available ev­i­dence, with no ex­tra weight for ar­gu­ments presently in front of one. Thus, they shouldn’t fluc­tu­ate dra­mat­i­cally back and forth; you should never be able to pre­dict which way your fu­ture prob­a­bil­ities will move.

  • If you can pre­dict what you’ll be­lieve a few years from now, con­sider be­liev­ing that already.

  • Re­mem­ber that if read­ing X-ist books will pre­dictably move your be­liefs to­ward X, and you know there are X-ist books out there, you should move your be­liefs to­ward X already. Re­mem­ber the Con­ser­va­tion of Ex­pected Ev­i­dence more gen­er­ally.

  • Con­sider what emo­tions are driv­ing the rapid fluc­tu­a­tions. If you’re un­com­fortable ever dis­agree­ing with your in­ter­locu­tors, build com­fort with dis­agree­ment. If you’re un­com­fortable not know­ing, so that you find your­self grasp­ing for one frame­work af­ter an­other, build your tol­er­ance for am­bi­guity, com­plex­ity, and un­knowns.

4. Up­date your near-mode an­ti­ci­pa­tions, not just your far-mode be­liefs.

Some­times your far-mode is smart and you near-mode is stupid. For ex­am­ple, Yvain’s ra­tio­nal­ist knows ab­stractly that there aren’t ghosts, but nev­er­the­less fears them. Other times, though, your near-mode is smart and your far-mode is stupid. You might “be­lieve” in an af­ter­life but re­tain a con­crete, near-mode fear of death. You might ad­vo­cate Com­mu­nism but have a sink­ing feel­ing in your stom­ach as you con­duct your tour of Stalin’s Rus­sia.

Thus: trust ab­stract rea­son­ing or con­crete an­ti­ci­pa­tions in differ­ent situ­a­tions, ac­cord­ing to their strengths. But, whichever one you bet your ac­tions on, keep the other one in view. Ask it what it ex­pects and why it ex­pects it. Show it why you dis­agree (vi­su­al­iz­ing your ev­i­dence con­cretely, if you’re try­ing to talk to your word­less an­ti­ci­pa­tions), and see if it finds your ev­i­dence con­vinc­ing. Try to grow all your cog­ni­tive sub­sys­tems, so as to form a whole mind.

5. Use raw mo­ti­va­tion, emo­tion, and be­hav­ior to de­ter­mine at least part of your pri­ori­ties.

One of the com­mon­est routes to the­ory-driven nut­ti­ness is to take a “goal” that isn’t your goal. Thus, folks claim to care “above all else” about their self­ish well-be­ing, the abo­li­tion of suffer­ing, an ob­jec­tive Mo­ral­ity dis­cov­er­able by su­per­in­tel­li­gence, or av­er­age util­i­tar­ian hap­piness-sums. They then find them­selves ei­ther with­out mo­ti­va­tion to pur­sue “their goals”, or else pul­led into chains of ac­tions that they dread and do not want.

Con­crete lo­cal mo­ti­va­tions are of­ten em­bar­rass­ing. For ex­am­ple, I find my­self con­cretely mo­ti­vated to “win” ar­gu­ments, even though I’d think bet­ter of my­self if I was driven by cu­ri­os­ity. But, like near-mode be­liefs, con­crete lo­cal mo­ti­va­tions can act as a safe­guard and an an­chor. For ex­am­ple, if you be­come ab­stractly con­fused about meta-ethics, you’ll still have a con­crete de­sire to pull ba­bies off train tracks. And so di­alogu­ing with your near-mode wants and mo­tives, like your near-mode an­ti­ci­pa­tions, can help build a ro­bust, trust-wor­thy mind.

Why it mat­ters (again)

Safety skills such as the above are worth learn­ing for three rea­sons.

  1. They help us avoid nutty ac­tions.

  2. They help us rea­son un­hesi­tat­ingly, in­stead of flinch­ing away out of fear.

  3. They help us build a ra­tio­nal­ity for the whole mind, with the strengths of near-mode as well as of ab­stract rea­son­ing.


[1] Th­ese are not the only rea­sons peo­ple fear think­ing. At min­i­mum, there is also:

  • Fear of so­cial cen­sure for the new be­liefs (e.g., for chang­ing your poli­tics, or failing to be­lieve your friend was jus­tified in his di­vorce);

  • Fear that part of you will use those new be­liefs to jus­tify ac­tions that you as a whole do not want (e.g., you may fear to read a study about up­sides of nico­tine, lest you use it as a ra­tio­nal­iza­tion to start smok­ing again; you may similarly fear to read a study about how eas­ily you can save Afri­can lives, lest it ends up prompt­ing you to donate money).

[2] Many points in this ar­ti­cle, and es­pe­cially in the “ex­plicit rea­son­ing is of­ten nuts” sec­tion, are stolen from Michael Vas­sar. Give him the credit, and me the blame and the up­votes.

[3] Carl points out that Eliezer points out that stud­ies show we can’t. But it seems like ex­plic­itly mod­el­ing when your friend is and isn’t ac­cu­rate, and when ex­plicit mod­els have and haven’t led you to good ac­tions, should at least help.