Rationality is about pattern recognition, not reasoning

Short ver­sion (cour­tesy of Nanashi)

Our brains’ pat­tern recog­ni­tion ca­pa­bil­ities are far stronger than our abil­ity to rea­son ex­plic­itly. Most peo­ple can rec­og­nize cats across con­texts with lit­tle men­tal ex­er­tion. By way of con­trast, ex­plic­itly con­struct­ing a for­mal al­gorithm that can con­sis­tently cats across con­texts re­quires great sci­en­tific abil­ity and cog­ni­tive ex­er­tion.

Very high level epistemic ra­tio­nal­ity is about re­train­ing one’s brain to be able to see pat­terns in the ev­i­dence in the same way that we can see pat­terns when we ob­serve the world with our eyes. Rea­son­ing plays a role, but a rel­a­tively small one. Suffi­ciently high qual­ity math­e­mat­i­ci­ans don’t make their dis­cov­er­ies through rea­son­ing. The math­e­mat­i­cal proof is the very last step: you do it to check that your eyes weren’t de­ceiv­ing you, but you know ahead of time that your eyes prob­a­bly weren’t de­ceiv­ing you.

I have a lot of ev­i­dence that this way of think­ing is how the most effec­tive peo­ple think about the world. I would like to share what I learned. I think that what I’ve learned is some­thing that lots of peo­ple are ca­pa­ble of learn­ing, and that learn­ing it would greatly im­prove peo­ple’s effec­tive­ness. But com­mu­ni­cat­ing the in­for­ma­tion is very difficult.

It took me 10,000+ hours to learn how to “see” pat­terns in ev­i­dence in the way that I can now. Right now, I don’t know how to com­mu­ni­cate how to do it suc­cinctly. In or­der to suc­ceed, I need col­lab­o­ra­tors who are open to spend a lot of time think­ing care­fully about the ma­te­rial, to get to the point of be­ing able to teach oth­ers. I’d wel­come any sug­ges­tions for how to find col­lab­o­ra­tors.

Long version

For most of my life, I be­lieved that epistemic ra­tio­nal­ity was largely about rea­son­ing care­fully about the world. I fre­quently ob­served peo­ple’s in­tu­itions lead­ing them astray. I thought that what differ­en­ti­ated peo­ple with high epistemic ra­tio­nal­ity is Carte­sian skep­ti­cism: the prac­tice of care­fully scru­ti­niz­ing all of one’s be­liefs us­ing de­duc­tive-style rea­son­ing.

When I met Holden Karnofsky, co-founder of GiveWell, I came to rec­og­nize that Holden’s gen­eral epistemic ra­tio­nal­ity was much higher than my own. Over the course of years of in­ter­ac­tion, I dis­cov­ered that Holden was not us­ing my style of rea­son­ing. In­stead, his be­liefs were backed by lots of in­de­pen­dent small pieces of ev­i­dence, which in ag­gre­gate sufficed to in­still con­fi­dence, even if no in­di­vi­d­ual piece of ev­i­dence was com­pel­ling by it­self. I fi­nally un­der­stood this in 2013, and it was a ma­jor epiphany for me. I wrote about it in two posts [1], [2].

After learn­ing data sci­ence, I re­al­ized that my “many weak ar­gu­ments” paradigm was also flawed: I had greatly over­es­ti­mated the role that rea­son­ing of any sort plays in ar­riv­ing at true be­liefs about the world.

In hind­sight, it makes sense. Our brains’ pat­tern recog­ni­tion ca­pa­bil­ities are far stronger than our abil­ity to rea­son ex­plic­itly. Most peo­ple can rec­og­nize cats across con­texts with lit­tle men­tal ex­er­tion. By way of con­trast, ex­plic­itly con­struct­ing a for­mal al­gorithm that can con­sis­tently cats across con­texts re­quires great sci­en­tific abil­ity and cog­ni­tive ex­er­tion. And the best al­gorithms that peo­ple have been con­structed (within the paradigm of deep learn­ing) are highly non­trans­par­ent: no­body’s been able to in­ter­pret their be­hav­ior in in­tel­ligible terms.

Very high level epistemic ra­tio­nal­ity is about re­train­ing one’s brain to be able to see pat­terns in the ev­i­dence in the same way that we can see pat­terns when we ob­serve the world with our eyes. Rea­son­ing plays a role, but a rel­a­tively small one. If one has de­vel­oped the ca­pac­ity to see in this way, one can con­struct post hoc ex­plicit ar­gu­ments for why one be­lieves some­thing, but these ar­gu­ments aren’t how one ar­rived at the be­lief.

The great math­e­mat­i­cian Henri Poin­care hinted at what I fi­nally learned, over 100 years ago. He de­scribed his ex­pe­rience dis­cov­er­ing a con­crete model of hy­per­bolic ge­om­e­try as fol­lows:

I left Caen, where I was liv­ing, to go on a ge­olog­i­cal ex­cur­sion un­der the aus­pices of the School of Mines. The in­ci­dents of the travel made me for­get my math­e­mat­i­cal work. Hav­ing reached Coutances, we en­tered an om­nibus to go to some place or other. At the mo­ment when I put my foot on the step, the idea came to me, with­out any­thing in my former thoughts seem­ing to have paved the way for it, that the trans­for­ma­tions I had used to define the Fuch­sian func­tions were iden­ti­cal with those of non-Eu­clidean ge­om­e­try. I did not ver­ify the idea; I should not have had time, as upon tak­ing my seat in the om­nibus, I went on with a con­ver­sa­tion already com­menced, but I felt a perfect cer­tainty. On my re­turn to Caen, for con­ve­nience sake, I ver­ified the re­sult at my leisure.”

Suffi­ciently high qual­ity math­e­mat­i­ci­ans don’t make their dis­cov­er­ies through rea­son­ing. The math­e­mat­i­cal proof is the very last step: you do it to check that your eyes weren’t de­ceiv­ing you, but you know ahead of time that your eyes prob­a­bly weren’t de­ceiv­ing you. Given that this is true even in math, which is thought of as the most log­i­cally rigor­ous sub­ject, it shouldn’t be sur­pris­ing that the same is true of epistemic ra­tio­nal­ity across the board.

Learn­ing data sci­ence gave me a deep un­der­stand­ing of how to im­plic­itly model the world in statis­ti­cal terms. I’ve crossed over into a zone of no longer know why I hold my be­liefs, in the same way that I don’t know how I per­ceive that a cat is a cat. But I know that it works. It’s rad­i­cally changed my life over a span of mere months. Amongst other things, I fi­nally iden­ti­fied a ma­jor blindspot that had un­der­pinned my near to­tal failure to achieve my goals be­tween ages 18 and 28.

I have a lot of ev­i­dence that this way of think­ing is how the most effec­tive peo­ple think about the world. Here I’ll give two ex­am­ples. Holden worked un­der Greg Jensen, the co-CEO of Bridge­wa­ter As­so­ci­ates, which is the largest hedge fund in the world. Carl Shul­man is one of the most epistem­i­cally ra­tio­nal mem­bers of the LW and EA com­mu­ni­ties. I’ve had a num­ber of very illu­mi­nat­ing con­ver­sa­tions with him, and in hind­sight, I see that he prob­a­bly thinks about the world in this way. See Luke Muehlhauser’s post Just the facts, ma’am! for hints of this. If I un­der­stand cor­rectly, Carl cor­rectly es­ti­mated Mark Zucker­berg’s fu­ture net worth as be­ing $100+ mil­lion upon meet­ing him as a fresh­man at Har­vard, be­fore Face­book.

I would like to share what I learned. I think that what I’ve learned is some­thing that lots of peo­ple are ca­pa­ble of learn­ing, and that learn­ing it would greatly im­prove peo­ple’s effec­tive­ness. But com­mu­ni­cat­ing the in­for­ma­tion is very difficult. Abel Prize win­ner Mikhail Gro­mov wrote

We are all fas­ci­nated with struc­tural pat­terns: pe­ri­od­ic­ity of a mu­si­cal tune, a sym­me­try of an or­na­ment, self-similar­ity of com­puter images of frac­tals. And the struc­tures already pre­pared within our­selves are the most fas­ci­nat­ing of all. Alas, most of them are hid­den from our­selves. When we can put these struc­tures-within-struc­tures into words, they be­come math­e­mat­ics. They are abom­inably difficult to ex­press and to make oth­ers un­der­stand.

It took me 10,000+ hours to learn how to “see” pat­terns in ev­i­dence in the way that I can now. Right now, I don’t know how to com­mu­ni­cate how to do it suc­cinctly. It’s too much for me to do as an in­di­vi­d­ual: as far as I know, no­body has ever been able to con­vey the rele­vant in­for­ma­tion to a siz­able au­di­ence!

In or­der to suc­ceed, I need col­lab­o­ra­tors who are open to spend a lot of time think­ing care­fully about the ma­te­rial, to get to the point of be­ing able to teach oth­ers. I’d wel­come any sug­ges­tions for how to find col­lab­o­ra­tors.