# Rationality is about pattern recognition, not reasoning

Short ver­sion (cour­tesy of Nanashi)

Our brains’ pat­tern recog­ni­tion ca­pa­bil­ities are far stronger than our abil­ity to rea­son ex­plic­itly. Most peo­ple can rec­og­nize cats across con­texts with lit­tle men­tal ex­er­tion. By way of con­trast, ex­plic­itly con­struct­ing a for­mal al­gorithm that can con­sis­tently cats across con­texts re­quires great sci­en­tific abil­ity and cog­ni­tive ex­er­tion.

Very high level epistemic ra­tio­nal­ity is about re­train­ing one’s brain to be able to see pat­terns in the ev­i­dence in the same way that we can see pat­terns when we ob­serve the world with our eyes. Rea­son­ing plays a role, but a rel­a­tively small one. Suffi­ciently high qual­ity math­e­mat­i­ci­ans don’t make their dis­cov­er­ies through rea­son­ing. The math­e­mat­i­cal proof is the very last step: you do it to check that your eyes weren’t de­ceiv­ing you, but you know ahead of time that your eyes prob­a­bly weren’t de­ceiv­ing you.

I have a lot of ev­i­dence that this way of think­ing is how the most effec­tive peo­ple think about the world. I would like to share what I learned. I think that what I’ve learned is some­thing that lots of peo­ple are ca­pa­ble of learn­ing, and that learn­ing it would greatly im­prove peo­ple’s effec­tive­ness. But com­mu­ni­cat­ing the in­for­ma­tion is very difficult.

It took me 10,000+ hours to learn how to “see” pat­terns in ev­i­dence in the way that I can now. Right now, I don’t know how to com­mu­ni­cate how to do it suc­cinctly. In or­der to suc­ceed, I need col­lab­o­ra­tors who are open to spend a lot of time think­ing care­fully about the ma­te­rial, to get to the point of be­ing able to teach oth­ers. I’d wel­come any sug­ges­tions for how to find col­lab­o­ra­tors.

Long version

For most of my life, I be­lieved that epistemic ra­tio­nal­ity was largely about rea­son­ing care­fully about the world. I fre­quently ob­served peo­ple’s in­tu­itions lead­ing them astray. I thought that what differ­en­ti­ated peo­ple with high epistemic ra­tio­nal­ity is Carte­sian skep­ti­cism: the prac­tice of care­fully scru­ti­niz­ing all of one’s be­liefs us­ing de­duc­tive-style rea­son­ing.

When I met Holden Karnofsky, co-founder of GiveWell, I came to rec­og­nize that Holden’s gen­eral epistemic ra­tio­nal­ity was much higher than my own. Over the course of years of in­ter­ac­tion, I dis­cov­ered that Holden was not us­ing my style of rea­son­ing. In­stead, his be­liefs were backed by lots of in­de­pen­dent small pieces of ev­i­dence, which in ag­gre­gate sufficed to in­still con­fi­dence, even if no in­di­vi­d­ual piece of ev­i­dence was com­pel­ling by it­self. I fi­nally un­der­stood this in 2013, and it was a ma­jor epiphany for me. I wrote about it in two posts [1], [2].

After learn­ing data sci­ence, I re­al­ized that my “many weak ar­gu­ments” paradigm was also flawed: I had greatly over­es­ti­mated the role that rea­son­ing of any sort plays in ar­riv­ing at true be­liefs about the world.

In hind­sight, it makes sense. Our brains’ pat­tern recog­ni­tion ca­pa­bil­ities are far stronger than our abil­ity to rea­son ex­plic­itly. Most peo­ple can rec­og­nize cats across con­texts with lit­tle men­tal ex­er­tion. By way of con­trast, ex­plic­itly con­struct­ing a for­mal al­gorithm that can con­sis­tently cats across con­texts re­quires great sci­en­tific abil­ity and cog­ni­tive ex­er­tion. And the best al­gorithms that peo­ple have been con­structed (within the paradigm of deep learn­ing) are highly non­trans­par­ent: no­body’s been able to in­ter­pret their be­hav­ior in in­tel­ligible terms.

Very high level epistemic ra­tio­nal­ity is about re­train­ing one’s brain to be able to see pat­terns in the ev­i­dence in the same way that we can see pat­terns when we ob­serve the world with our eyes. Rea­son­ing plays a role, but a rel­a­tively small one. If one has de­vel­oped the ca­pac­ity to see in this way, one can con­struct post hoc ex­plicit ar­gu­ments for why one be­lieves some­thing, but these ar­gu­ments aren’t how one ar­rived at the be­lief.

The great math­e­mat­i­cian Henri Poin­care hinted at what I fi­nally learned, over 100 years ago. He de­scribed his ex­pe­rience dis­cov­er­ing a con­crete model of hy­per­bolic ge­om­e­try as fol­lows:

I left Caen, where I was liv­ing, to go on a ge­olog­i­cal ex­cur­sion un­der the aus­pices of the School of Mines. The in­ci­dents of the travel made me for­get my math­e­mat­i­cal work. Hav­ing reached Coutances, we en­tered an om­nibus to go to some place or other. At the mo­ment when I put my foot on the step, the idea came to me, with­out any­thing in my former thoughts seem­ing to have paved the way for it, that the trans­for­ma­tions I had used to define the Fuch­sian func­tions were iden­ti­cal with those of non-Eu­clidean ge­om­e­try. I did not ver­ify the idea; I should not have had time, as upon tak­ing my seat in the om­nibus, I went on with a con­ver­sa­tion already com­menced, but I felt a perfect cer­tainty. On my re­turn to Caen, for con­ve­nience sake, I ver­ified the re­sult at my leisure.”

Suffi­ciently high qual­ity math­e­mat­i­ci­ans don’t make their dis­cov­er­ies through rea­son­ing. The math­e­mat­i­cal proof is the very last step: you do it to check that your eyes weren’t de­ceiv­ing you, but you know ahead of time that your eyes prob­a­bly weren’t de­ceiv­ing you. Given that this is true even in math, which is thought of as the most log­i­cally rigor­ous sub­ject, it shouldn’t be sur­pris­ing that the same is true of epistemic ra­tio­nal­ity across the board.

Learn­ing data sci­ence gave me a deep un­der­stand­ing of how to im­plic­itly model the world in statis­ti­cal terms. I’ve crossed over into a zone of no longer know why I hold my be­liefs, in the same way that I don’t know how I per­ceive that a cat is a cat. But I know that it works. It’s rad­i­cally changed my life over a span of mere months. Amongst other things, I fi­nally iden­ti­fied a ma­jor blindspot that had un­der­pinned my near to­tal failure to achieve my goals be­tween ages 18 and 28.

I have a lot of ev­i­dence that this way of think­ing is how the most effec­tive peo­ple think about the world. Here I’ll give two ex­am­ples. Holden worked un­der Greg Jensen, the co-CEO of Bridge­wa­ter As­so­ci­ates, which is the largest hedge fund in the world. Carl Shul­man is one of the most epistem­i­cally ra­tio­nal mem­bers of the LW and EA com­mu­ni­ties. I’ve had a num­ber of very illu­mi­nat­ing con­ver­sa­tions with him, and in hind­sight, I see that he prob­a­bly thinks about the world in this way. See Luke Muehlhauser’s post Just the facts, ma’am! for hints of this. If I un­der­stand cor­rectly, Carl cor­rectly es­ti­mated Mark Zucker­berg’s fu­ture net worth as be­ing \$100+ mil­lion upon meet­ing him as a fresh­man at Har­vard, be­fore Face­book.

I would like to share what I learned. I think that what I’ve learned is some­thing that lots of peo­ple are ca­pa­ble of learn­ing, and that learn­ing it would greatly im­prove peo­ple’s effec­tive­ness. But com­mu­ni­cat­ing the in­for­ma­tion is very difficult. Abel Prize win­ner Mikhail Gro­mov wrote

We are all fas­ci­nated with struc­tural pat­terns: pe­ri­od­ic­ity of a mu­si­cal tune, a sym­me­try of an or­na­ment, self-similar­ity of com­puter images of frac­tals. And the struc­tures already pre­pared within our­selves are the most fas­ci­nat­ing of all. Alas, most of them are hid­den from our­selves. When we can put these struc­tures-within-struc­tures into words, they be­come math­e­mat­ics. They are abom­inably difficult to ex­press and to make oth­ers un­der­stand.

It took me 10,000+ hours to learn how to “see” pat­terns in ev­i­dence in the way that I can now. Right now, I don’t know how to com­mu­ni­cate how to do it suc­cinctly. It’s too much for me to do as an in­di­vi­d­ual: as far as I know, no­body has ever been able to con­vey the rele­vant in­for­ma­tion to a siz­able au­di­ence!

In or­der to suc­ceed, I need col­lab­o­ra­tors who are open to spend a lot of time think­ing care­fully about the ma­te­rial, to get to the point of be­ing able to teach oth­ers. I’d wel­come any sug­ges­tions for how to find col­lab­o­ra­tors.

• I’d be glad to offer what help I can. Based on other posts of yours, I would definitely prac­tice brevity. This post is over 1000 words long and eas­ily could be con­densed to 250 or less.

• Per our email ex­change, here is the con­densed ver­sion that uses only your origi­nal writ­ing:

“Our brains’ pat­tern recog­ni­tion ca­pa­bil­ities are far stronger than our abil­ity to rea­son ex­plic­itly. Most peo­ple can rec­og­nize cats across con­texts with lit­tle men­tal ex­er­tion. By way of con­trast, ex­plic­itly con­struct­ing a for­mal al­gorithm that can con­sis­tently cats across con­texts re­quires great sci­en­tific abil­ity and cog­ni­tive ex­er­tion.

Very high level epistemic ra­tio­nal­ity is about re­train­ing one’s brain to be able to see pat­terns in the ev­i­dence in the same way that we can see pat­terns when we ob­serve the world with our eyes. Rea­son­ing plays a role, but a rel­a­tively small one. Suffi­ciently high qual­ity math­e­mat­i­ci­ans don’t make their dis­cov­er­ies through rea­son­ing. The math­e­mat­i­cal proof is the very last step: you do it to check that your eyes weren’t de­ceiv­ing you, but you know ahead of time that it’s your eyes prob­a­bly weren’t de­ceiv­ing you.

I have a lot of ev­i­dence that this way of think­ing is how the most effec­tive peo­ple think about the world. I would like to share what I learned. I think that what I’ve learned is some­thing that lots of peo­ple are ca­pa­ble of learn­ing, and that learn­ing it would greatly im­prove peo­ple’s effec­tive­ness. But com­mu­ni­cat­ing the in­for­ma­tion is very difficult.

It took me 10,000+ hours to learn how to “see” pat­terns in ev­i­dence in the way that I can now. Right now, I don’t know how to com­mu­ni­cate how to do it suc­cinctly. In or­der to suc­ceed, I need col­lab­o­ra­tors who are open to spend a lot of time think­ing care­fully about the ma­te­rial, to get to the point of be­ing able to teach oth­ers. I’d wel­come any sug­ges­tions for how to find col­lab­o­ra­tors.”

Notes:

• I re­moved all the quo­ta­tions. Although I’m guess­ing they were prob­a­bly key to your own un­der­stand­ing of the is­sue, I don’t think they are an effi­cient way to im­prove other peo­ple’s un­der­stand­ing.

• Much of the post was ded­i­cated (un­nec­es­sar­ily) to why your view­point is right rather than just stat­ing your view­point. Peo­ple who agree with you don’t need to be con­vinced. Peo­ple who dis­agree with you aren’t go­ing to be swayed by your ar­gu­ments.

• I re­moved a few para­graphs that re­peated them­selves.

• While I agree that there’s value to be­ing able to state the sum­mary of the view­point, I can’t help but feel that brevity is the wrong ap­proach to take to this sub­ject in par­tic­u­lar. If the point is that effec­tive peo­ple rea­son by ex­am­ples and see­ing pat­terns rather than by ma­nipu­lat­ing log­i­cal ob­jects and func­tions, then re­mov­ing the ex­am­ples and pat­terns to just leave log­i­cal ob­jects and func­tions is be­tray­ing the point!

Some­what more gen­er­ally, yes, there is value in tel­ling peo­ple things, but they need to be ex­plained if you want to com­mu­ni­cate with peo­ple that don’t already un­der­stand them.

• I definitely agree that you shouldn’t be so brief as to not get your point across, I think the level of brevity de­pends on what your goal is. In this case, he’s ask­ing for help. It isn’t un­til 1,500 words in that the two most im­por­tant ques­tions: “What does he want?” and “Why should I help him?” are an­swered.

(Be­sides, he speci­fi­cally wanted help in com­mu­ni­cat­ing things suc­cinctly.)

• The post re­minded me of The cre­ative mind by Mar­garet Bow­den; her ex­am­ples, in par­tic­u­lar Kekule see­ing the ben­zene ring, seem rele­vant here. (Although the book definitely could be shorter:)

• Here is the even-fur­ther ed­ited ver­sion, con­densed to 150 words.

I have a lot of ev­i­dence that the most effec­tive peo­ple in the world have a very spe­cific way of think­ing. They use their brain’s pat­tern-match­ing abil­ities to pro­cess the world, rather than us­ing ex­plicit rea­son­ing.

Our brain can pat­tern match much more effi­ciently than it can rea­son. Most peo­ple can rec­og­nize a cat very eas­ily. But cre­at­ing an al­gorithm to rec­og­nize cats is far more difficult. And break­throughs of any kind are very rarely made via ex­plicit rea­son­ing, but rather through a com­plex and rapid-fire com­bi­na­tion of ideas.

Do­ing this is some­thing that many peo­ple are ca­pa­ble of learn­ing. But, it took me 10,000+ hours to learn how to “see” the world way that I can now, and I do not know how to com­mu­ni­cate this pro­cess suc­cinctly. In or­der to help peo­ple, I need col­lab­o­ra­tors who are will­ing to help clar­ify my thoughts. I’d wel­come any sug­ges­tions.

You’ll note it very quickly gets to the three main points:

• What are you talk­ing about?

• Why should we listen to you?

• What do you want?

Let me know if I sum­ma­rized any part of your thoughts in­cor­rectly.

• Thanks very much, both for the shorted ver­sion and for the notes. I added the shorted ver­sion at the top of my post.

• Not a prob­lem at all. What you’re talk­ing about is some­thing I be­lieve in, so I’m glad to help.

• I do not think the en­tire post was too long, but I do think read­ing the short ver­sion first was helpful. It’s sort of like read­ing an ab­stract be­fore div­ing into a jour­nal ar­ti­cle. If noth­ing else, it helps peo­ple who are un­in­ter­ested save some time.

Peo­ple who agree with you don’t need to be con­vinced. Peo­ple who dis­agree with you aren’t go­ing to be swayed by your ar­gu­ments.

I’m not con­vinced this is true, but re­gard­less, what about peo­ple who nei­ther agree nor dis­agree? To a large ex­tent, ex­plain­ing why your view­point is right is ex­actly the same thing as ex­plain­ing in de­tail what your view­point is.

• An in­ter­est­ing post. You started with the as­sump­tion that for­mal rea­son­ing is the right way to go and found out that it’s not nec­es­sar­ily so. Let me start from the op­po­site end: the ob­ser­va­tion that the great ma­jor­ity of peo­ple rea­son all the time by pat­tern-match­ing, this is the nor­mal, de­fault, bog-stan­dard way of figur­ing things out.

You do not need to “re­train” peo­ple to think in pat­terns—they do so nat­u­rally.

Look­ing at my­self, I cer­tainly do think in terms of pat­terns—in­ter­nal maps and struc­tures. Typ­i­cally I carry a more-or-less co­her­ent map of the sub­ject in my head (which cer­tain ar­eas be­ing fuzzy or in­com­plete, that’s fine) and the map is kinda-spa­tial. When a new piece of data comes in, I try to fit it into the ex­ist­ing (in my head) struc­ture and see if it’s a good fit. If it’s not a good fit, it’s like a peb­ble in a shoe—an ir­ri­tant and an ob­vi­ous prob­lem. The prob­lem is fixed ei­ther by rein­ter­pret­ing the data and its im­pli­ca­tions, or by bend­ing and ad­just­ing the struc­ture so there is a proper place for the new data nugget. Some­times both hap­pen.

For­mal rea­son­ing is atyp­i­cal for me, that’s why I’m not that good at math. I find situ­a­tions where you have enough hard data to for­mally rea­son about it to be un­usual and rare (that would prob­a­bly be differ­ent if I were an en­g­ineer or an ac­coun­tant :-D). Most of­ten you have stochas­tic rea­son­ing with prob­a­bil­ity dis­tri­bu­tions and con­di­tional out­comes and that is amenable to anal­y­sis only at low lev­els. At high enough lev­els you’re ba­si­cally back to pat­tern recog­ni­tions, ideally with some sup­port from for­mal rea­son­ing.

In any case, I’m not sure why do you think that teach­ing peo­ple to think in pat­terns will be hard or will lead to ma­jor jumps in pro­duc­tivity. Peo­ple already do this, all the time. Essen­tially you are talk­ing about un­lear­ing the re­li­ance on for­mal­ism which is ap­pli­ca­ble to very few.

• The refer­ence class that I’ve im­plic­itly had in mind in writ­ing my post is math­e­mat­i­ci­ans /​ LWers /​ EAs, who do seem to think in the way that I had been. See my post Many weak ar­gu­ments and the typ­i­cal mind.

Peo­ple out­side of this refer­ence gen­er­ally use im­plicit statis­ti­cal mod­els that are not so great. For such peo­ple, the po­ten­tial gains come from learn­ing how to build much bet­ter im­plicit statis­ti­cal mod­els (as I did as a re­sult of my ex­po­sure to data sci­ence.) I don’t know whether learn­ing more ad­vanced statis­tics would work for you per­son­ally—but for me, it was what I needed. His­tor­i­cally, most peo­ple who have very good im­plicit statis­ti­cal mod­els seem to have learned by ob­serv­ing oth­ers who do. But it can be hard to get ac­cess to them (e.g. I would not have been able to con­nect with Greg Jensen, Holden’s former boss, dur­ing my early 20′s, as Holden did.)

• math­e­mat­i­ci­ans /​ LWers /​ EAs

Math­e­mat­i­ci­ans, yes, but that’s kinda nat­u­ral be­cause peo­ple be­come good math­e­mat­i­ci­ans pre­cisely by the virtue of be­ing very good at for­mal rea­son­ing. But I don’t know about LW/​EA in gen­eral—I doubt most of them have “math­e­mat­i­cal minds”.

Peo­ple out­side of this refer­ence gen­er­ally use im­plicit statis­ti­cal mod­els that are not so great.

Really? Math geeks/​LW/​EA are the creme de la creme, the ul­ti­mate in­tel­lec­tual elite? I haven’t no­ticed. “Nor­mal” peo­ple cer­tainly don’t have great think­ing skills. But there is a very large num­ber of smart and highly suc­cess­ful peo­ple who are out­side of your refer­ence class. They greatly out­num­ber the math/​LW/​EA crowd.

• But I don’t know about LW/​EA in gen­eral—I doubt most of them have “math­e­mat­i­cal minds”.

Within the LW cluster I’ve seen a lot of fo­cus on pre­ci­sion. It’s not un­com­mon for peo­ple in the com­mu­nity to miss the main points that I’m try­ing to make in fa­vor of fo­cus­ing on a sin­gle sen­tence that I wrote that seems wrong. I have sel­dom had this ex­pe­rience in con­ver­sa­tion with peo­ple out­side of the LW cluster: my con­ver­sa­tion part­ners out­side of the LW cluster gen­er­ally hold my view: that it’s in­evitably the case that one will say some things things that are wrong, and that it’s best to fo­cus on the main points that some­one is try­ing to make.

Really? Math geeks/​LW/​EA are the creme de la creme, the ul­ti­mate in­tel­lec­tual elite? I haven’t no­ticed. “Nor­mal” peo­ple cer­tainly don’t have great think­ing skills. But there is a very large num­ber of smart and highly suc­cess­ful peo­ple who are out­side of your refer­ence class. They greatly out­num­ber the math/​LW/​EA crowd.

By “gen­er­ally” I meant “most peo­ple,” not “for a fixed per­son” – i.e. I don’t nec­es­sar­ily dis­agree with you.

Separately, I be­lieve that a large frac­tion of trans­fer­able hu­man cap­i­tal is in fact in elite math and physics, but that’s a long con­ver­sa­tion. My im­pres­sion is that good physi­cists do use the style of think­ing that I just learned. In the case of elite math­e­mat­i­ci­ans, I think it would take like 5 years of get­ting up to speed with real world stuff be­fore their strength as thinkers started to come out vividly.

• great ma­jor­ity of people

Of peo­ple wor­ld­wide, or of peo­ple read­ing this post? Con­sid­er­ing the former leads to this failure mode.

• Of peo­ple wor­ld­wide, or of peo­ple read­ing this post?

Both.

Math­e­mat­i­ci­ans are weird peo­ple, they think differ­ently :-) I don’t think most of LW is math­e­mat­i­ci­ans.

• As a math­e­mat­i­cian I can tes­tify that even most math­e­mat­i­ci­ans think in maps.

• I agree that a pic­ture of many weak ar­gu­ments sup­port­ing or un­der­min­ing ex­plicit claims does not cap­ture whet hu­mans do—the in­fer­ences them­selves are much more com­plex than log­i­cal de­duc­tions, such that we don’t yet know any way of rep­re­sent­ing the ac­tual ob­jects that are be­ing ma­nipu­lated. I think this is the main­stream view, cer­tainly in AI now.

I don’t know what it means to say that our pat­tern recog­ni­tion ca­pa­bil­ities are stronger than our log­i­cal rea­son­ing; they are two differ­ent kinds of cog­ni­tive tasks. It seems like say­ing that we are much bet­ter at run­ning fast than lift­ing heavy ob­jects. Some­times you can do a task in one way or the other, and we might say that one or the other is bet­ter way to get some­thing done. And we can com­pare to other an­i­mals, or to ma­chines, and talk about com­par­a­tive ad­van­tage. And so on.

Per­haps the most rele­vant claim would be ask­ing what frac­tion of var­i­ance in out­comes is de­scribed by one char­ac­ter­is­tic or an­other, or in which do­main is prac­tice most helpful. I think that ex­plicit prin­ci­ples about how to rea­son do not dis­t­in­guish good math­e­mat­i­ci­ans from each other, though they may dis­t­in­guish math­e­mat­i­ci­ans at differ­ent times. The situ­a­tion seems similar in most en­deav­ors. I think this is be­cause it is so much eas­ier to trans­fer ex­plicit in­for­ma­tion be­tween peo­ple, so the resi­d­ual is what differ­en­ti­ates. Learn­ing the ex­plicit info is still the right first step to mas­tery, though it’s not most of the work.

Im­prov­ing ex­plicit info and norms seems to be the main way that we progress as a civ­i­liza­tion, be­cause that’s what you can build on, share, and so on. But of course, those can be ex­plicit norms about how to train any part of rea­son­ing, and I am pretty ag­nos­tic about what kind of rea­son­ing is best to try to im­prove.

Over­all I feel like there is a broad ver­sion of your the­sis that I think is clearly true. You are no doubt mak­ing a more spe­cific claim. I’m in­ter­ested to see it fleshed out. I can eas­ily see dis­agree­ing with it. If I do, it’s prob­a­bly be­cause I have a broader sense of /​ dis­tri­bu­tion over ways we might use our brain. For ex­am­ple:

“Very high level epistemic ra­tio­nal­ity is about re­train­ing one’s brain to be able to see pat­terns in the ev­i­dence in the same way that we can see pat­terns when we ob­serve the world with our eyes.”

Is a spe­cific way we can use our built in pat­tern recog­ni­tion, one could list per­haps half a dozen similar tac­tics at a similar level of ab­strac­tion. I will be in­ter­ested to see if you have ev­i­dence that dis­t­in­guishes be­tween these prin­ci­ples and sin­gles out one as most im­por­tant. (You could also in­ter­pret your quote in a broad way, in which case I think it is very clear. So I guess I should just wait to dis­cuss!)

Note that the ac­cu­racy of this kind of literal anal­ogy, or how far you can take that anal­ogy, is a ques­tion that re­searchers in com­puter sci­ence and AI ex­plic­itly dis­cuss. Most ev­ery­one agrees that there is some of this, but I think there is le­gi­t­i­mate un­cer­tainty about how much.

Also, when you say “don’t make their dis­cov­er­ies through rea­son­ing,” it’s not ex­actly clear what this means. This might also be some­thing that I dis­agree with strongly, or it may be some­thing that I think is very clear (or maybe some­where in be­tween). Logic plays a key role in the math­e­mat­i­cian’s rea­son­ing. None of the steps the math­e­mat­i­cian makes are valid in­fer­ences in any for­mal proof sys­tem (but nei­ther are any steps of most pub­lished proofs!), though of­ten she will write or think pairs of state­ments that are in fact re­lated to each other in a log­i­cally pre­cise way, and the ex­is­tence of such re­la­tion­ships is a key part of why the cog­ni­tive pro­cess yields cor­rect re­sults.

• 27 May 2015 7:37 UTC
5 points

This is in­ter­est­ing. I have found that when you are like 16, you of­ten want ev­ery­thing to be su­per log­i­cal and ev­ery­thing that is not feels stupid. And grow­ing up largely means ac­cept­ing “com­mon sense”, which at the end of the day means rely­ing more on pat­tern recog­ni­tion. (This is also poli­ti­cally rele­vant—young rad­i­cal­ism is of­ten about match­ing ev­ery­thing with a log­i­cal sound­ing ide­ol­ogy, while peo­ple when they grow and be­come more mod­er­ate sim­ply care about what typ­i­cal pat­terns tend to re­sult in hu­man flour­ish­ing more than about ide­ol­ogy.)

There is some­thing in pat­tern recog­ni­tion that feels pretty “con­ser­va­tive” in the not-too-poli­ti­cal sense. Log­i­cal rea­son­ing is an in­di­vi­d­u­al­is­tic thing, you can make your own philos­o­phy of things es­pe­cially if you are young and feel you are so much smarter than ev­ery­body else. But if you treat your brain as a pat­tern sen­sor, it is differ­ent.

First of all ex­pe­rience mat­ters more than sheer bright­ness. You start to think less and less that old peo­ple are dinosauric fools and re­spect them more and more. (Of course the right kind of ex­pe­rience mat­ters more than just clock­ing in a lot of birth­days. There are 19 years old guys who are so fa­natic about cars that they spend day and night work­ing on them and prob­a­bly know why your car does not run well than your dad does.)

Se­cond, throw­ing a lot of brains on the pat­terns i.e. ac­tu­ally listen­ing to other peo­ple’s opinions starts to look good. It scales differ­ently. You can feel you can out-rea­son and out-logic a thou­sand peo­ple be­cause logic is not ad­di­tive. But be­ing a sen­sor its. When hunt­ing for a de­tail, you can­not out-see two thou­sand eye­balls. So you start to re­spect other peo­ple’s opinions more.

Third, there are de­pos­i­to­ries of rec­og­nized pat­terns. They are usu­ally called best prac­tices, ac­cepted prac­tices or even tra­di­tions. They start to mat­ter.

It is a very sober­ing ex­pe­rience, and for me it was kind of painful (be­cause hu­mil­i­at­ing, deflat­ing), it hap­pened be­tween 21 and 26.

It is turn­ing peo­ple more con­ser­va­tive in the not-so-poli­ti­cal sense and it is prob­a­bly a good thing, at least I think it made me bet­ter off, al­though it was painful. For ex­am­ple, in ar­chi­tec­ture, do you value bravely tra­di­tion buck­ing origi­nal de­sign, or you value tra­di­tional pat­tern-book ar­chi­tec­ture? Scru­ton ar­gues the later is more likely to cre­ate an en­vi­ron­ment in which peo­ple feel good.

• Much of what you say res­onates with me. I think that a ma­jor prob­lem that very smart young peo­ple of­ten have is not meet­ing older coun­ter­parts of them­selves. The great math­e­mat­i­cian Don Zagier was an ex­treme prodigy, pro­gress­ing so rapidly that he earned his PhD at age 20. But de­spite the fact that he pos­sessed im­mense in­nate abil­ity, he needed to learn from a great math­e­mat­i­cian in or­der to be­come one. He wrote

My first real teacher was in my third year of grad­u­ate school, Friedrich Hirze­bruch. It was through him that I be­gan to think like a real math­e­mat­i­cian. This is some­thing you can’t teach your­self but have to learn from a mas­ter.

There’s some over­lap be­tween what you write and Nick Beck­stead’s post Com­mon sense as a prior, which I recom­mend if you haven’t read it be­fore.

I went through some­thing similar to what you went through, but for me it has a happy end­ing – it’s not that my ideas were wrong all along, it’s that I hadn’t yet learned how to in­te­grate them with the wis­dom of peo­ple who were older than me. I sus­pect that some­thing similar is true of you to some de­gree as well.

• I have found that when you are like 16, you of­ten want ev­ery­thing to be su­per log­i­cal and ev­ery­thing that is not feels stupid. And grow­ing up largely means ac­cept­ing “com­mon sense”, which at the end of the day means rely­ing more on pat­tern recog­ni­tion.

For a coun­terex­am­ple, I am 16 and al­most all my de­ci­sions/​per­cep­tions are based on im­plicit pat­tern recog­ni­tion more than ex­plicit rea­son­ing.

ETA: I think I missed your point.

• Very high level epistemic ra­tio­nal­ity is about re­train­ing one’s brain to be able to see pat­terns in the ev­i­dence in the same way that we can see pat­terns when we ob­serve the world with our eyes.

Can you ex­plain a bit more why you think that the way peo­ple with very high level epistemic ra­tio­nal­ity pro­cess ev­i­dence is analo­gous to how we rec­og­nize vi­sual pat­terns? Do you think these two men­tal pro­cesses are fun­da­men­tally us­ing the same al­gorithms, or just that both are sub­con­scious com­pu­ta­tions that we don’t un­der­stand very well?

• I’m go­ing on the one learn­ing al­gorithm hy­poth­e­sis that’s be­come pop­u­lar in parts of the neu­ro­science com­mu­nity, to­gether with my sub­jec­tive im­pres­sions. In­tu­itively, it seems par­si­mo­nious to sup­pose that evolu­tion hi­jacked the al­gorithms used for sen­sory pro­cess­ing for gen­eral in­tel­li­gence. I don’t think that the al­gorithms are the same, but I would guess that they’re rel­a­tives.

• In­ter­est­ing. I found one pa­per that ex­plains the one learn­ing al­gorithm hy­poth­e­sis and gives ev­i­dence for it. Quot­ing from it:

There seems to be some ev­i­dence for a sin­gle al­gorithm ex­plain­ing the com­pu­ta­tions performed by the pri­mary au­di­tory, vi­sual, mo­tor, and so­matosen­sory cor­tices. Given how lit­tle is known about higher-level pro­cess­ing im­me­di­ately down­stream of these pri­mary ar­eas, it is pre­ma­ture to gen­er­al­ize to other ar­eas lo­cated in the oc­cip­i­tal lobe. Less is known about the de­tails of the com­pu­ta­tions performed in the ar­eas lo­cated in an­te­rior cor­tex and, in par­tic­u­lar, the pre­frontal cor­tex, which is dis­pro­por­tionately en­larged in hu­mans when com­pared to non-hu­man pri­mates.

Is there any­thing more up to date or com­pre­hen­sive than this pa­per?

This tan­gent aside, I agree that it would be re­ally valuable to im­prove the way we pro­cess ev­i­dence sub­con­sciously. I’m a bit skep­ti­cal that you’ve ac­tu­ally found such a method, but I hope that you suc­ceed in writ­ing it down and that it re­ally works.

• Our Co­a­lesc­ing Minds pa­per had the one learn­ing al­gorithm hy­poth­e­sis as one of its as­sump­tions; I wasn’t the neu­ro­science ex­pert, but my co-au­thor was, and here’s what he wrote about that premise (note that the pa­per was in­tended for a rel­a­tively pop­u­lar au­di­ence, so the neu­ro­science de­tail was kept light):

An adult hu­man neo­cor­tex con­sists of sev­eral ar­eas which are to vary­ing de­grees spe­cial­ized to pro­cess differ­ent types of in­for­ma­tion. The func­tional spe­cial­iza­tion is cor­re­lated with the anatom­i­cal differ­ences of differ­ent cor­ti­cal ar­eas. Although there are ob­vi­ous differ­ences be­tween ar­eas, most cor­ti­cal ar­eas share many func­tional and anatom­i­cal traits. There has been con­sid­er­able de­bate on whether cor­ti­cal micro­cir­cuits are di­verse or canon­i­cal [Bux­ho­eve­den & Casanova, 2002; Nel­son, 2002] but we ar­gue that the differ­ences are vari­a­tions of the same un­der­ly­ing cor­ti­cal al­gorithm, rather than en­tirely differ­ent al­gorithms. This is be­cause most cor­ti­cal ar­eas seem to have the ca­pa­bil­ity of pro­cess­ing any type of in­for­ma­tion. The differ­ences seem to be a mat­ter of op­ti­miza­tion to a spe­cific type of in­for­ma­tion, rather than a differ­ent un­der­ly­ing prin­ci­ple.

The cor­ti­cal ar­eas do lose much of their plas­tic­ity dur­ing mat­u­ra­tion. For in­stance, it is pos­si­ble to lose one’s abil­ity to see col­ors if a spe­cific vi­sual cor­ti­cal area re­spon­si­ble for color vi­sion is dam­aged. The adult brain is not plas­tic enough to com­pen­sate for this dam­age, as the rele­vant re­gions have already spe­cial­ized to their tasks. If the same brain re­gions were to be dam­aged dur­ing early child­hood, color blind­ness would most likely not re­sult.

How­ever, this lack of plas­tic­ity re­flects learn­ing and spe­cial­iza­tion dur­ing the lifes­pan of the brain rather than in­nate al­gorith­mic differ­ences be­tween differ­ent cor­ti­cal ar­eas. Plenty of ev­i­dence sup­ports the idea that the differ­ent cor­ti­cal ar­eas can pro­cess any spa­tiotem­po­ral pat­terns. For in­stance, the cor­ti­cal area which nor­mally re­ceives au­di­tory in­for­ma­tion and de­vel­ops into the au­di­tory cor­tex will de­velop vi­sual rep­re­sen­ta­tions if the ax­ons car­ry­ing au­di­tory in­for­ma­tion are sur­gi­cally re­placed by ax­ons car­ry­ing vi­sual in­for­ma­tion from the eyes [New­ton & Sur, 2004]. The ex­per­i­ments were car­ried out with young kit­tens, but a some­what similar sen­sory sub­sti­tu­tion is seen even in adult hu­mans: re­lay­ing vi­sual in­for­ma­tion through a tac­tile dis­play mounted on the tongue will re­sult in vi­sual per­cep­tion [Vuillerme & Cuisiner, 2009]. What first feels like tick­ling in the tongue will start feel­ing like see­ing. In other words, the ex­pe­rience of see­ing is not in the vi­sual cor­tex but in the struc­ture of the in­com­ing in­for­ma­tion.

Another ex­am­ple of the mam­malian brain’s abil­ity to pro­cess any type of in­for­ma­tion is the de­vel­op­ment of trichro­matic vi­sion in mice that, like mam­malian an­ces­tors, nor­mally have a dichro­matic vi­sion [Ja­cobs et al., 2007]. All it takes for a mouse to de­velop pri­mate-like color vi­sion is the ad­di­tion of a gene en­cod­ing the pho­topig­ment which evolved in pri­mates. When mice are born with this ex­tra gene, their cor­tex is able to adapt to the new source in­for­ma­tion from the retina and to make sense of it. Even the adult cor­ti­cal ar­eas of hu­mans can be sur­pris­ingly adap­tive as long as the changes hap­pen slowly enough [Feuillet et al., 2007]. Fi­nally, Marzullo et al. [2010] demon­strated that rats im­planted with elec­trodes both in their mo­tor and vi­sual cor­tices can learn to mod­u­late the out­put from their mo­tor cor­tex based on feed­back given to vi­sual cor­tex.

• The pa­per you linked to about the one learn­ing al­gorithm hy­poth­e­sis is from 2012. Since that time the the­ory has gained sig­nifi­cant strength from the ad­vances in DL, and in par­tic­u­lar the work on deep re­in­force­ment learn­ing. Prov­ing that an ANN with a rel­a­tively sim­ple ini­tial/​prior ar­chi­tec­ture and about 1 mil­lion neu­rons can reach hu­man-level perfor­mance on a set of 100 games when trained end to end with RL is pretty strong (albeit in­di­rect) ev­i­dence for the one learn­ing hy­poth­e­sis.

One key re­main­ing ques­tion is then: how does the brain ac­tu­ally im­ple­ment ap­prox­i­mate op­ti­miza­tion/​learn­ing that is at least as good as back-prop? We know that back-prop is not biolog­i­cally re­al­is­tic. On that front, Ben­gio’s group has made sig­nifi­cant re­cent progress with a new tech­nique/​the­ory called tar­get prop­a­ga­tion 1, which origi­nated in part as an ex­pla­na­tion for how the brain could im­ple­ment credit as­sign­ment, but it also shows promise as a po­ten­tial re­place­ment for back­prop 2 - which fur­ther in­creases the biolog­i­cal plau­si­bil­ity.

In terms of more di­rect ev­i­dence, the hip­pocam­pus in par­tic­u­lar ap­pears to have a sim­ple ex­pla­na­tion in terms of re­in­force­ment learn­ing 3.

In terms of the pre­frontal cor­tex in par­tic­u­lar, there are work­ing the­o­ries that ex­plain much of the PFC as a set of mod­ules spe­cial­ized for work­ing mem­ory buffers that are con­trol­led by gat­ing units in the basal gan­glia. That whole sys­tem in par­tic­u­lar is also driven/​learned through dopamine based RL.

• You have not un­der­stood cor­rectly re­gard­ing Carl. He claimed, in hind­sight, that Zucker­berg’s po­ten­tial could’ve been dis­t­in­guished in fore­sight, but he did not do so.

• I’m puz­zled, is there a way to read his comment

Peo­ple de­scribed him to me as re­sem­bling a young Bill Gates. His es­ti­mated ex­pected fu­ture wealth based on that data if pur­su­ing en­trepreneur­ship, and in­formed by the data about the re­la­tion­ship of all of the char­ac­ter­is­tics I could track with it, was in the 9-figure range. Then add in that face­book was a very promis­ing startup (I did some mar­ket siz­ing es­ti­mates for it, and peo­ple who looked at it and its early re­sults were re­li­ably im­pressed).

other than as him do­ing it at the time?

• Yes, as his post facto ar­gu­ment.

• 29 May 2015 12:15 UTC
3 points

If I un­der­stand cor­rectly, Carl cor­rectly es­ti­mated Mark Zucker­berg’s fu­ture net worth as be­ing \$100+ mil­lion upon meet­ing him as a fresh­man at Har­vard, be­fore Face­book.

Well, if I un­der­stand the post cor­rectly, even as a fresh­man, Mark ap­par­ently had pre­vi­ous ex­pe­rience with own­ing/​run­ning a busi­ness, and was de­liber­ately try­ing to be­come a tech en­trepreneur. Now, given that some­one is from a priv­ileged fam­ily, is at­tend­ing school at (al­most) the max­i­mally priv­ileged and well-con­nected in­sti­tu­tion (at least on the East Coast) for wannabe rich guys, has pre­vi­ous ex­pe­rience with busi­ness by the time he reaches age 18, pos­sesses enough in­tel­li­gence to be go­ing to school at Har­vard (which, de­spite be­ing partly a priv­ilege club, still re­quires gen­uine in­tel­li­gence and work-ethic to get into), and is clearly driven to be­come a rich tech guy… it’s not that un­re­al­is­tic to say that he would, even­tu­ally, achieve his goal.

It just re­quires con­di­tion­ing on a bunch of facts that most peo­ple don’t know about Mark Zucker­berg. But once you do con­di­tion on the facts that were available at the time… it all adds up to nor­mal­ity.

But any­way, to ad­dress the heart of the post… “pat­tern-match­ing” and in­duc­tive rea­son­ing are similar but not iden­ti­cal. AFAIK, the hu­man mind performs prob­a­bil­is­tic causal in­duc­tion on what­ever data “makes it through” the sen­sory cor­tices, which are quite pos­si­bly but not surely do­ing some­thing like in­de­pen­dent com­po­nent anal­y­sis via un­su­per­vised neu­ral learn­ing… so yeah.

Most peo­ple have some solid in­tu­itions about how prob­a­bil­is­tic causal in­duc­tion works, not in quan­ti­ta­tive terms but in terms of “What hap­pens if I do this?”. That’s the whole rea­son TVTropes ex­ists. The prob­lem is that we tell peo­ple only spe­cial lit­tle causal mod­els called “logic” or “de­bate rules” or, God help the poor vic­tims, “philos­o­phy” are in­vested with the Spe­cial Epistemic Nor­ma­tive Power of tel­ling you true things, rather than speci­fi­cally train­ing the in­duc­tive fac­ul­ties we re­ally rely on, the fac­ulty of spot­ting what re­al­ity is do­ing by look­ing.

• I’d wel­come any sug­ges­tions for how to find col­lab­o­ra­tors.

Keep post­ing the ma­te­rial here. Post to Main. Don’t worry about it not be­ing pol­ished enough: you’ll get plenty of feed­back. Ig­nore feed­back that isn’t use­ful to you.

• Thanks for the post Jonah.

In med­i­cal school, I was taught that when you’re a novice doc­tor, you’ll make di­ag­noses and plans us­ing de­liber­a­tive rea­son­ing, but that ex­perts even­tu­ally pat­tern-match ev­ery­thing.

If that’s true, then pat­tern-match­ing might arise nat­u­rally with ex­pe­rience, or it might be some­thing that’s difficult to achieve in many do­mains at once.

When I read your ar­ti­cle, the rea­sons that I might doubt that you de­serve col­lab­o­ra­tors are:

1) that en­thu­si­as­tic self-re­ports of spe­cial per­cep­tual-cog­ni­tive abil­ities have a low prior prob­a­bil­ity 2) that you lack mechanis­tic ex­pla­na­tions of things you did that led to you per­ciev­ing some types of pat­terns more easily

Also, it seems al­most like a par­ody that you spent 10,000+ hours learn­ing to see pat­terns in ev­i­dence. Like, I thought you might in the next line say ‘gotcha, that’s de­scribing how I learnt English as a child’! Do you mean you learnt pat­terns, or you learnt how to learn pat­terns? And if you want oth­ers to help you to teach the ma­te­rial, you’ll al­most cer­tainly need to start by teach­ing pieces of it your­self!

Put­ting all of this to the side, I en­joyed the post—thanks for writ­ing it—and am in­ter­ested to see what you come up with.

• Some con­trary ev­i­dence about use­ful­ness of ex­plicit mod­els: http://​​www.busi­ness­in­sider.com/​​elon-musk-first-prin­ci­ples-2015-1

My take is that you need both, some things are un­der­stood bet­ter “from first prin­ci­ples” (en­g­ineer­ing) oth­ers are more suit­able for pat­tern match­ing (poli­tics).

• Yes, as I say in an­other com­ment, my sense had been that what works best is 50% in­tu­ition and 50% ex­plicit rea­son­ing, and now I think it’s more like 95% vs 5%. If you’re spend­ing all of your time think­ing, that still leaves roughly an hour a day for ex­plicit rea­son­ing, which is sub­stan­tially more than usu­ally.

• I think there might be some con­fu­sion over terms here. I don’t think “pat­tern match­ing” is the best way to phrase this.

Musk seems to be ar­gu­ing for “rule learn­ing” (figur­ing out the un­der­ly­ing rule) as op­posed to “ex­am­ple learn­ing” (in­ter­po­lat­ing to the near­est ex­am­ple in your col­lec­tion). In the book Make it Stick, the au­thors men­tion that rule learn­ers tend to be bet­ter learn­ers. (Th­ese terms come from the psy­cholog­i­cal liter­a­ture.)

I don’t think this ob­ser­va­tion is in­com­pat­i­ble with the im­por­tance of rec­og­niz­ing pat­terns. You need to “pat­tern match” which rule to in­voke. You also need to rec­og­nize the pat­tern that is the rule in the first place. Rec­og­niz­ing which ex­am­ples to use also could be pat­tern match­ing, too, so this is why I don’t think the term is right.

In the same book men­tioned pre­vi­ously, the au­thors write about Kah­ne­man’s sys­tems 1 and 2, and I got the im­pres­sion that mas­tery of­ten is mov­ing things from sys­tem 2 (more care­ful rea­son­ing) to sys­tem 1 (au­to­matic pat­tern match­ing, which might sim­ply be pre­com­puted). Here’s an ex­am­ple: Vaniver sug­gested to me be­fore that (if I re­call cor­rectly) when play­ing chess, some­one might not ex­plic­itly con­sider a cer­tain num­ber of moves; their brain just has a map that goes from the cur­rent state of the board and other in­for­ma­tion to their next move. Devel­op­ing this abil­ity re­quires rec­og­niz­ing the right pat­terns in the game, which could come from sim­ply hav­ing a large library of ex­am­ples to in­ter­po­late from, or what­not. This is pre­cisely what I thought of when I read that it took (the fa­mous) 10,000 hours for Jon­ahSinick to see the pat­terns.

(To be fair, you do need both, but it seems that if you can de­velop good rules, you should use them. Also, de­vel­op­ing ac­cu­rate in­tu­ition is use­ful, whether it uses ex­plicit rules or not.)

• Musk is very in­ter­est­ing in his re­gard. He didn’t start SpaceX and Tesla be­cause he rea­soned him­self into those pro­jects hav­ing a high chance of com­mer­cial suc­cess.

He choose them be­cause he be­lieved in those goals. He’s driven by pas­sion to­wards those goals.

• Even if I agree with you on the goals (I can claim he used meta-ra­tio­nal­ity here, in the sense that some­one should try to make hu­mans in­ter­plane­tary species, even if he thought his chance of suc­cess was less than 50%) a lot the think­ing that made him ar­rive at SpaceX seemed to be “one can ac­tu­ally do this way cheaper than the cur­rently ac­cepted stan­dards, based on cost of ma­te­ri­als etc”

• I don’t think Jonah or I ar­gues that you should never make calcu­la­tions. Musks did make many de­ci­sions on that path and from the out­side it’s hard to get an overview of what drives which de­ci­sion.

• What you are de­scribing is my na­tive way of think­ing. My mind fits large amounts of in­for­ma­tion to­gether into an aes­thetic whole. I took me a while to figure out that other peo­ple don’t think this way, and they can’t eas­ily just ab­sorb pat­terns from ev­i­dence.

This mode of think­ing has been de­scribed as In­tro­verted Think­ing in Ben Kovitz’s ob­scure psy­chol­ogy wiki about Lenore Thom­son’s ob­scure take on Jun­gian psy­chol­ogy. Some of you are fa­mil­iar with Jun­gian func­tions through MBTI, the My­ers-Briggs Type Indi­ca­tor. In­tro­verted Think­ing (ab­bre­vi­ated Ti) is the dom­i­nant func­tion of the INTP type.

It will only take a few quotes to illus­trate why you are talk­ing about the same thing:

In­tro­verted Think­ing (Ti) is the at­ti­tude that be­neath the com­plex­ity of what is man­i­fest (ap­par­ent, ob­served, ex­pe­rienced) there is an un­der­ly­ing unity: a source or essence that emerges and takes form in differ­ent ways de­pend­ing on cir­cum­stances. What is man­i­fest is seen as a man­i­fes­ta­tion of some­thing. From a Ti stand­point, the way to re­spond to things is in a way that is faith­ful to that un­der­ly­ing cause or source and helps it emerge fully and com­plete, with­out in­terfer­ence from any no­tion of self. The way to un­der­stand that un­der­ly­ing essence is to learn to si­mul­ta­neously see many re­la­tion­ships within what is man­i­fest, to see ev­ery el­e­ment in re­la­tion to ev­ery other el­e­ment, the re­la­tion­ships be­ing the “sig­na­ture” of the un­der­ly­ing unity. This can only be ex­pe­rienced di­rectly, not sec­ond-hand.

In­tro­verted think­ing is a form of men­tal rep­re­sen­ta­tion in which ev­ery in­put, ev­ery vari­able, ev­ery as­pect of things is con­sid­ered si­mul­ta­neously and holis­ti­cally to per­ceive causal, math­e­mat­i­cal, and aes­thetic or­der. What you know by Ti, you know with your hands, your eyes, your mus­cles, even a tingling sen­sa­tion “down­stairs” be­cause you sense that ev­ery­thing fits. Every vari­able is fair game to vary, ev­ery com­bi­na­tion of vari­ables wor­thy of con­sid­er­a­tion; the only ul­ti­mate ar­biter is how well the parts form a unified whole rather than a jum­ble.

In­tro­verted Think­ing (Ti) is con­trasted with Ex­traverted Think­ing (Te):

From the Te per­spec­tive, any­thing for which you can’t give an op­er­a­tional defi­ni­tion in terms of mea­sure­ment (an “ob­jec­tive test”) doesn’t ex­ist. The de­ci­sion crite­ria are defined not ex­actly in terms of the things: they’re defined in terms of ob­ser­va­tions of a sort that any­one can do and get the same re­sult. You put the to­tal­ity of the real-world situ­a­tion onto your scales, so that all causal fac­tors come into play—both known and un­known. What’s ac­cessible to you is the read­ing on the scale: that and only that is the ba­sis for your de­ci­sion.

As a dom­i­nant func­tion, Te typ­i­cally leads one to pur­sue and col­lect re­li­able ways of mak­ing de­ci­sions to get pre­dictable re­sults. The re­peata­bil­ity of a pro­cess be­comes one of the main crite­ria for find­ing it valuable. Re­peat­able pro­cesses are valuable from a Te per­spec­tive be­cause they en­able you to make agree­ments with other peo­ple, where there is no doubt as to whether each party has fulfilled its part of the agree­ment. Mak­ing and de­liv­er­ing on promises is of­ten how a Te at­ti­tude leads one to un­der­stand ethics.

From the Ti stand­point, com­mu­ni­ca­tion is pos­si­ble only be­tween peo­ple who share some com­mon ex­pe­rience of the things that they’re talk­ing about. To say some­thing that you can un­der­stand, I need to re­late it log­i­cally to things in your own ex­pe­rience. To show you how far a piece of wood bends, in­stead of giv­ing a nu­mer­i­cal mea­sure (Te), I’d ei­ther en­courage you to bend a piece of wood your­self, or find some math­e­mat­i­cally similar thing that you know about and re­late wood-bend­ing to that. Words can­not be defined prior to the re­al­ity that they’re about; words and crite­ria defined in­de­pen­dently of the re­al­ity would be mean­ingless. The world it­self pro­vides a nat­u­ral set of refer­ence points, aris­ing from the real, causal struc­ture of things. Ul­ti­mately, to talk is to say, “I mean *that).”

In­tro­verted Think­ing uses lan­guage and con­cepts merely as poin­t­ers to pat­terns in re­al­ity that are in­cred­ibly more com­plex than any­thing that can be de­scribed in words. In con­trast, Ex­traverted Think­ing is about step-by-step jus­tifi­ca­tion ac­cord­ing to shared lan­guage and critera. A com­mon failure mode of Ex­traverted Think­ing is King on The Moun­tain, which I think ev­ery­one will in­stantly rec­og­nize.

In­tro­verted Think­ing and Ex­traverted Think­ing, along with Ex­traverted In­tu­ition and In­tro­verted In­tu­ition, are com­bined to cre­ate ra­tio­nal­ity. Ex­traverted In­tu­ition pro­vides the idea gen­er­a­tion, In­tro­verted Think­ing pro­vides pat­tern recog­ni­tion, Ex­traverted Think­ing han­dles jus­tifi­ca­tion, and In­tro­verted In­tu­ition avoids bias. Ac­cord­ing to the Jung-Thom­son-Kovitz the­ory, all of these modes of think­ing provide benefits and failure modes. For ex­am­ple, a failure mode of In­tro­verted Think­ing is that since it is aes­thetic and sub­jec­tive, it can be very hard for In­tro­verted Thinkers with differ­ent in­puts to rec­on­cile wor­ld­views if they differ, whereas Ex­traverted Thinkers could slowly ham­mer out agree­ment step-by-step.

LessWrong seems mostly dom­i­nated by INTJs, who have In­tro­verted In­tu­ition and Ex­traverted Think­ing. They are mostly fo­cused on jus­tifi­ca­tion and bias. Th­ese are im­por­tant skills, but In­tro­verted Think­ing is im­por­tant for mar­shal­ing the pri­ors of the to­tal­ity of your ex­pe­rience.

• Con­tin­u­ing a bit…

It’s truly strange see­ing you say some­thing like “Very high level epistemic ra­tio­nal­ity is about re­train­ing one’s brain to be able to see pat­terns in the ev­i­dence in the same way that we can see pat­terns when we ob­serve the world with our eyes.” I already com­pul­sively do the thing you talk­ing about train­ing your­self to do! I can’t stop see­ing pat­terns. I don’t claim that the pat­terns I see are always true, just that’s it’s re­ally easy for me to see them.

For me, think­ing is like a gale wind car­ry­ing puz­zle pieces that dance in the air and as­sem­ble them­selves in front of me in gi­gan­tic struc­tures, with­out any in­ter­ven­tion by me. I do not ex­pe­rience this as an “abil­ity” that I could “train”, be­cause it doesn’t feel like there is any sort of “me” that is do­ing it: I am merely the pas­sive ob­server. “Train­ing” pat­tern recog­ni­tion sounds as strange to me train­ing vi­sion it­self: all I have to do is open my eyes, and it hap­pens. Ap­par­ently it isn’t that way for ev­ery­one?

The only ways I’ve dis­cov­ered to train my pat­tern recog­ni­tion is to feed my­self more in­for­ma­tion of higher qual­ity (be­cause garbage-in, garbage out), and to train my at­ten­tion. Once I can learn to no­tice some­thing, I will start to com­pul­sively see pat­terns in it. For some­one who isn’t com­pul­sively max­ing out their pat­tern recog­ni­tion already, maybe it’s train­able.

Another ex­am­ple: my brain is of­ten lin­ing peo­ple in rows of 3 or 4 ac­cord­ing to some col­lec­tion of traits. There might “some­thing” where Alice has more of it than Bob, and Bob has more of it than Carol. I see them stand­ing next to each other, kind of like pieces on a chess­board. Ba­si­cally, I think what my brain is do­ing is some kind of fac­tor anal­y­sis where it is iden­ti­fy­ing un­named di­men­sions be­hind peo­ple’s per­son­al­ities and us­ing them to make pre­dic­tions. I’m pretty sure that not ev­ery­one is con­stantly do­ing this, but I could be wrong.

Per­haps some­one smarter than me might be able to vi­su­al­ize a larger num­ber of peo­ple in mul­ti­ple di­men­sions in peo­ple-space. That would be pretty cool.

On a triv­ial level, ev­ery­one can do pat­tern-recog­ni­tion to some de­gree, merely by virtue of be­ing a hu­man with gen­eral in­tel­li­gence. Yet some peo­ple can syn­the­size larger amounts of in­for­ma­tion col­lected over a longer pe­riod of time, up­date their syn­the­sis faster and more fre­quently, and can draw qual­i­ta­tively differ­ent sorts of con­nec­tions.

I think that’s what you are get­ting at when you talk about pat­tern recog­ni­tion be­ing im­por­tant for epistemic ra­tio­nal­ity. Pat­tern recog­ni­tion is like a men­tal mus­cle: some peo­ple have it stronger, some peo­ple have differ­ent types of mus­cles, and it’s prob­a­bly train­able. There is only one sort of de­duc­tion, but per­haps there are many ap­proaches to in­duc­tion.

Luke’s de­scrip­tion of Carl Shul­man re­minds me of Ben Kovitz’s de­scrip­tion of In­tro­verted Think­ing as con­stantly writ­ing and rewrit­ing a book. When you ask Carl Shul­man a ques­tion on AI, and he starts giv­ing you facts in­stead of a straight an­swer, he is re­veal­ing part of his book.

“Many weak ar­gu­ments” is not how this feels from the in­side. From the in­side, it all feels like one ar­gu­ment. Ex­cept the thing you are hear­ing from Carl Shul­man is re­ally only the tip of the ice­berg be­cause he can­not talk fast enough. His real an­swer to your ques­tion in­volves the to­tal­ity of his knowl­edge of AI, or per­haps the to­tal­ity of the con­tents of his brain.

For an­other ex­am­ple of tak­ing ar­gu­ments in to­tal­ity vs. in iso­la­tion, see King On The Moun­tain, de­scribing an im­ma­ture form of Ex­traverted Think­ing:

In the King-on-the-Moun­tain style of con­ver­sa­tion, one per­son (the King) makes a provoca­tive state­ment, and re­quires that oth­ers re­fute it or ad­mit to be­ing wrong. The King is the judge of whether any at­tempted re­fu­ta­tion is suc­cess­ful.

A re­fu­ta­tion, for the King to rule it valid, must be com­pletely self-con­tained: the King will ig­nore any­thing out­side the way he has con­cep­tu­al­ized the topic (see be­low for an ex­tended illus­tra­tion). If the re­fu­ta­tion in­volves two or more propo­si­tions that must be heard to­gether, the King will rule that each propo­si­tion in­di­vi­d­u­ally fails to re­fute his state­ment. He won’t ad­dress mul­ti­ple propo­si­tions taken to­gether. Once a propo­si­tion is re­jected, he gives it no fur­ther con­sid­er­a­tion. A re­fu­ta­tion must meet the King’s pre-ex­ist­ing crite­ria and make sense in terms of the King’s pre-ex­ist­ing way of un­der­stand­ing the sub­ject. The King will rule any sug­ges­tion that his crite­ria are not pro­duc­ing in­sight as an at­tempt to cheat. […] The amount of in­for­ma­tion that the King con­sid­ers at one time is very small: one state­ment. He makes one de­ci­sion at a time. He then moves on to the next at­tempted re­fu­ta­tion, putting all pre­vi­ous de­ci­sions be­hind him. The broad panorama—of math­e­mat­i­cal, spa­tial, and tem­po­ral re­la­tion­ships be­tween many facts—that makes up the pro-evolu­tion ar­gu­ment, which need to be viewed all at once to be per­sua­sive, can­not get in, un­less some­one finds a way to pack­age it as a one-step-at-a-time ar­gu­ment (and the King has pa­tience to hear it). Where his op­po­nent was at­tempt­ing to com­mu­ni­cate just one idea, the King heard many sep­a­rate ideas to be judged one by one.

Some of the failure modes of In­tro­verted Think­ing in­volves see­ing imag­i­nary pat­terns, deal­ing with cor­rupted in­put, or hav­ing aes­thetic bi­ases (aes­thetic bias is when you are bi­ased to­wards an ex­pla­na­tion that look neat or har­mo­nious). Com­mu­ni­ca­tion is also hard, but your true ar­gu­ments would take a book to de­scribe, if they could even be put into words at all.

• How many bad ideas or am­bigu­ously true ideas do math­e­mat­i­ci­ans have for ev­ery good idea they pro­duce? How many peo­ple feel “deep cer­tain­ties” about hy­pothe­ses that never pan out? Even when some­times cor­rect, do their hunches gen­er­ally do bet­ter than chance alone would sug­gest? I agree with the idea that pat­tern recog­ni­tion is im­por­tant, but think your claims are go­ing too far. My opinion is that suc­cess­ful pat­tern recog­ni­tion, even in the hands of the best hu­man ex­perts, re­lies heav­ily on ex­plicit rea­son­ing that takes con­trol over the recog­ni­tion mechanisms and keeps them ac­cu­rately tar­geted. Without cum­ber­some re­straints that re­sist men­tal ma­nipu­la­tions, hu­mans are more likely to in­vent nu­merol­ogy than Calcu­lus. Fil­ter­ing out bad ideas or chains of thought that pat­tern recog­ni­tion brings into one’s head is im­por­tant.

A sig­nifi­cant rea­son I’ve had prob­lems with ad­vanced Calcu­lus is that my brain starts in­vent­ing too many jus­tifi­ca­tions for things, and then I be­come un­able to dis­t­in­guish be­tween re­mem­bered rules which are valid and ones which my mind in­vented with­out suffi­cient jus­tifi­ca­tion. The differ­ence be­tween a su­per­sti­tion, a heuris­tic, and a rule is ex­tremely im­por­tant, but I don’t think pat­tern recog­ni­tion is well equipped to mon­i­tor thoughts to main­tain these dis­tinc­tions. I see pat­tern recog­ni­tion as be­ing about what things have in com­mon. That has a lot to recom­mend it, but differ­ences are im­por­tant too. I wouldn’t say ei­ther pat­tern recog­ni­tion or rea­son­ing are of pri­mary im­por­tance. They’re two halves of a whole, ei­ther alone is al­most use­less while both to­gether can be very very strong. In my own case, it’s the re­stric­tions I find difficult, be­ing imag­i­na­tive is al­most too easy for me.

• A sig­nifi­cant rea­son I’ve had prob­lems with ad­vanced Calcu­lus is that my brain starts in­vent­ing too many jus­tifi­ca­tions for things, and then I be­come un­able to dis­t­in­guish be­tween re­mem­bered rules which are valid and ones which my mind in­vented with­out suffi­cient jus­tifi­ca­tion.

That may re­flect more of a lack of suffi­cient prac­tice on your part than any­thing else. It takes a long time to be­come fa­mil­iar enough with a topic that your brain can start in­tu­itively and spon­ta­neously gen­er­at­ing good ideas on that topic. As an ex­am­ple, de­spite hav­ing spent sev­eral years play­ing chess, I still have to con­sider ev­ery po­si­tion care­fully and with de­liber­a­tion; al­though there have been cases in which the move which im­me­di­ately springs to mind is cor­rect, I’ve found that in gen­eral the op­po­site is true. How­ever, there is ev­i­dence that top grand­mas­ters do not view chess po­si­tions this way; their play is based a lot more on “feel­ing” than “think­ing”. (I don’t have the source for it, but I definitely re­mem­ber read­ing some­thing about it in both GEB and Think­ing, Fast and Slow.) Clearly, this means that de­spite hav­ing played chess for so long, I have still not yet reached the level at which in­tu­ition can play a sig­nifi­cant role in my calcu­la­tions. Based on what you’ve writ­ten here, I would judge it likely that you are in a similar situ­a­tion with re­spect to calcu­lus.

(Also see this. I think that the “post-rigor­ous” stage de­scribed in this post matches nicely with what Jonah said above.)

• Thanks :-). I was go­ing to re­spond along these lines be­fore see­ing that you had spo­ken my mind.

• If you’re right, in chess it re­quires years and years of do­main spe­cific prac­tice to get pat­tern recog­ni­tion skills ad­e­quately pre­pared so that scrupu­lous thought is not re­quired when eval­u­at­ing moves. That doesn’t seem like an ar­gu­ment against the im­por­tance of scrupu­lous thought to me, it seems like the op­po­site. Scrupu­lous thought is very hard to avoid rely­ing on.

I think you’re wrong how­ever. I think once you reach a cer­tain level of fa­mil­iar­ity with a sub­ject, the dis­tinc­tion be­tween pat­tern recog­ni­tion and scrupu­lous rea­son­ing it­self breaks down. I don’t think chess ex­perts only use the raw pro­cess­ing power of their sub­con­scious minds when eval­u­at­ing the board, I think they al­ter­nate be­tween mak­ing bot­tom-up as­sess­ments and top-down judge­ments. The ac­counts given in the neu­rol­ogy books are re­ac­tions to the pop­u­lar per­cep­tion that rea­son­ing abil­ities are all that mat­ters in chess, but if they’ve given you the im­pres­sion that rea­son­ing isn’t im­por­tant in chess then I feel like they may have gone too far in em­pha­siz­ing their point. Ex­pert chess play­ers cer­tainly feel like they’re do­ing some­thing im­por­tant with their con­scious minds. They give nar­ra­tive de­scrip­tions of their rounds reg­u­larly. I ac­knowl­edge that ex­plicit thought is not all there is to play­ing chess, but I’m not pre­pared to say ex­perts’ ac­counts of their thoughts are just ego­ist delu­sions, or any­thing like that.

I sup­pose one point I’m try­ing to make here is that bi­ased stupid thought and ge­nius in­sight­ful thought feel the same from the in­side. And I think even ge­niuses have bi­ased stupid thoughts of­ten, even within their fields of ex­per­tise, and so the im­por­tance of rigor should not be down­played even for them. Ge­nius isn’t a qual­ity for avoid­ing bad thoughts, it’s qual­ity that makes some­one ca­pa­ble of hav­ing a few good thoughts in ad­di­tion to all their other bad ones. When ge­nius is paired with good filters, then it pro­duces ex­cel­lence reg­u­larly. Without good filters, it’s much less re­li­able.

Fi­nally, when you’re deal­ing with the­o­ries about the uni­verse the situ­a­tion is differ­ent than when deal­ing with strat­egy games. You can’t make a dumb sub­ar­gu­ment and then a smart sub­ar­gu­ment and have the two state­ments com­bine to pro­duce a mod­er­ately valuable hy­poth­e­sis. If you start driv­ing down the wrong street, cor­rectly fol­low­ing the rest of a list of di­rec­tions will not be helpful to you. Ri­gor is im­por­tant through­out all steps of the en­tire pro­cess. No mis­takes can lead to suc­cess with­out first be­ing un­done (or at least al­most none will—there are always ex­cep­tions).

• I think even ge­niuses have bi­ased stupid thoughts of­ten, even within their fields of ex­per­tise, and so the im­por­tance of rigor should not be down­played even for them.

To use the chess anal­ogy once more: this seems to con­flict with the fact that in chess, top grand­mas­ters’ in­tu­itions are al­most always cor­rect (and the rare ex­cep­tions al­most always in­volve some ab­surd-look­ing move that only gets found af­ter the fact through post-game com­puter anal­y­sis). Quite of­ten, you’ll see a chess au­thor tout­ing the im­por­tance of “quiet judg­ment” in­stead of “brute calcu­la­tion”; that sug­gests ex­tremely strongly to me that most grand­mas­ters don’t calcu­late out ev­ery move—and for good rea­son: it would be ex­haust­ing!

Like­wise, I’m given to un­der­stand many math­e­mat­i­ci­ans also have this sort of in­tu­itive judg­ment; of course, it takes a long time to build up the nec­es­sary back­ground knowl­edge and brain con­nec­tions for such judg­ment, but then, Jonah never claimed oth­er­wise. From the post it­self:

It took me 10,000+ hours to learn how to “see” pat­terns in ev­i­dence in the way that I can now. Right now, I don’t know how to com­mu­ni­cate how to do it suc­cinctly. It’s too much for me to do as an in­di­vi­d­ual: as far as I know, no­body has ever been able to con­vey the rele­vant in­for­ma­tion to a siz­able au­di­ence!

If we could find a way to quickly build up the type of judg­ment de­scribed above, it could very well change the way peo­ple do things for­ever, but alas, we’re not quite there. That’s the whole point of Jonah’s re­quest for col­lab­o­ra­tion. (In an ideal world, I’d par­ti­ci­pate, but as a 17-year-old I doubt I’d have much to con­tribute and a lot of my time is used up prepar­ing for col­lege at this stage any­way, so… yeah. Un­for­tu­nate.)

• I was not aware most grand­mas­ters’ first in­stincts ended up be­ing cor­rect usu­ally, in­ter­est­ing.

Like­wise, I’m given to un­der­stand many math­e­mat­i­ci­ans also have this sort of in­tu­itive judg­ment; of course, it takes a long time to build up the nec­es­sary back­ground knowl­edge and brain con­nec­tions for such judg­ment, but then, Jonah never claimed oth­er­wise. From the post it­self:

I’ve been chang­ing my po­si­tion some­what though­out this con­ver­sa­tion, just so it’s clear. At this point, I guess what I think is that a hard dis­tinc­tion be­tween “rea­son­ing” and “pat­tern recog­ni­tion” doesn’t make much sense. It seems like suc­cess­ful pat­tern recog­ni­tion is to a sig­nifi­cant ex­tent com­prised of scrupu­lously rea­soned ideas that have been in­ter­nal­ized. If some­one hy­po­thet­i­cally re­fused to use ex­plicit rea­son­ing while be­ing taught to rec­og­nize cer­tain pat­terns, I’d ex­pect that per­son to have a more difficult time learn­ing. Rea­son­ing about ideas in the way that is slow and de­liber­a­tive even­tu­ally makes pat­terns eas­ier to rec­og­nize in the way that is fast and in­tu­itive. If some­one doesn’t in­cor­po­rate slow thought origi­nated re­stric­tions into their fast pat­tern match­ing ca­pa­bil­ities, then they will start be­liev­ing in faces that ap­pear in the clouds, as­sum­ing that they ever learn to pat­tern match at all.

• Without cum­ber­some re­straints that re­sist men­tal ma­nipu­la­tions, hu­mans are more likely to in­vent nu­merol­ogy than Calcu­lus.

That is true which is why most peo­ple are not great thinkers. How­ever high skill might not come from ex­plicit rea­son­ing, but from re­fin­ing the pat­tern match­ing to prune away false branches. Mastery of a skill comes not from the abil­ity to do a lot of Bayesian up­dates cor­rectly and re­ally fast, it comes from prac­tic­ing till your in­tu­ition (=pat­tern-recog­ni­tion en­g­ine) starts to re­li­ably lead you to­wards good solu­tions and away from bad ones.

• I be­lieve that what you are de­scribing is known as in­ter­nal­iz­ing.

• 28 May 2015 19:17 UTC
1 point

Do you re­ally think that this is some­thing that can be taught through writ­ing?

Most in­tu­itive pat­tern recog­ni­tion comes through re­peated prac­tice, and I think that it might make more sense to cre­ate some sort of train­ing reg­i­men/​coach­ing that al­lows oth­ers to have that prac­tice, in­stead of writ­ing a post about it.

If you did cre­ate this train­ing, I’d be in­cred­ibly in­ter­ested in tak­ing it (prob­a­bly up to about \$300 or so, which is ad­mit­tedly small for this type of thing).

• al­gorithms that peo­ple have been con­structed (within the paradigm of deep learn­ing) are highly non­trans­par­ent: no­body’s been able to in­ter­pret their be­hav­ior in in­tel­ligible terms.

Not quite true Jonah: http://​​arxiv.org/​​pdf/​​1311.2901.pdf

• Even if what I said isn’t liter­ally true, it’s still true that the cut­ting edge re­search in pat­tern recog­ni­tion is in deep learn­ing, where the al­gorithms that are in some sense highly non­trans­par­ent.

• Upon read­ing your com­ment about non-trans­parency in DL I thought of the ex­act same pa­per on vi­su­al­iz­ing ANN fea­tures that Dr_Man­hat­tan posted. There was a re­cent post on the ma­chine learn­ing sub­red­dit about us­ing similar tech­niques to in­ves­ti­gate the neu­ral rep­re­sen­ta­tions au­to­mat­i­cally learned in lan­guage model RNNs.

There is no in­trin­sic con­nec­tion be­tween trans­parency and au­to­matic fea­ture learn­ing tech­niques. Con­sider the case of a large re­search team where the work in cre­at­ing a vi­sual sys­tem is di­vided amongst dozens of re­searchers, who each cre­ate spe­cific fea­tures for cer­tain lay­ers/​mod­ules. The re­sult­ing fea­tures are not in­trin­si­cally opaque just be­cause the lead re­searcher doesn’t nec­es­sar­ily un­der­stand the de­tails of each fea­ture each en­g­ineer came up with. The lead re­searcher sim­ply needs to in­vest the time in un­der­stand­ing those fea­tures (if so de­sired).

Deep learn­ing sim­ply au­to­mates the te­dious fea­ture en­g­ineer­ing pro­cess. You can always in­ves­ti­gate the fea­tures or spe­cific cir­cuits the ma­chine came up with—if so de­sired. It is true that ML and DL op­ti­miza­tion tools in par­tic­u­lar are of­ten used as black boxes where the re­searcher doesnt know or care about the de­tails of the solu­tion—but that does not im­ply that the al­gorithms them­selves are in­trin­si­cally opaque.

• Does this cap­ture any of what you’re talk­ing about? This is my in­tu­itive take away from the post so I want to check if it’s not what is in­tended. An anal­ogy: we know that the lens has flaws and we can learn spe­cific moves to shift the lens a bit so that we can see the flaws more eas­ily. For those with high lev­els of epistemic ra­tio­nal­ity, bump­ing the lens around in just the right ways is, or has be­come, an au­to­matic pro­cess such that they seem to have a magic abil­ity to always catch the flaws right away. We ask them for an al­gorithm to do that and they point to a mish­mash of differ­ent ways of nudg­ing the lens. Both sides feel mildly silly. “So I move the it 2cm to the left fol­lowed by a 2 de­gree ro­ta­tion clock­wise?”, “Uh, just nudge it around a bit dude, you should see what I mean af­ter try­ing for a while.”

• Yes :-). I wouldn’t say that it perfectly en­cap­su­lates what I was try­ing to say, but I my­self don’t yet know how to give a perfect en­cap­su­la­tion ei­ther. Some of the com­ments that other com­menters have made are very much on point as well.

• Is this what you were refer­ring to in “Is Scott Alexan­der bad at math?” when you said that be­ing good at math is largely about “aes­thetic dis­cern­ment” rather than “in­tel­li­gence”? Be­cause if so that seems like an un­usual no­tion of “in­tel­li­gence”, to use it to mean ex­plicit rea­son­ing only and ex­clude pat­tern recog­ni­tion. Like it would seem very odd to say “MIT Mys­tery Hunt doesn’t re­quire much in­tel­li­gence,” even if fre­quently do­main knowl­edge is more im­por­tant to spot­ting its pat­terns.

Or did you mean some­thing else? I re­al­ize this is not the same post, but I’m just not clear on how you’re sep­a­rat­ing “aes­thetic dis­cern­ment” from “in­tel­li­gence” here; the sort of aes­thetic dis­cern­ment needed for math­e­mat­ics seems like a kind of in­tel­li­gence.

• The dis­tinc­tion that I’m draw­ing is that in­tel­li­gence is about the ca­pac­ity to rec­og­nize pat­terns whereas aes­thetic dis­cern­ment is about se­lec­tively be­ing drawn to­ward pat­terns that are im­por­tant. I be­lieve that in­tel­li­gence ex­plains a large frac­tion of the var­i­ance in math­e­mat­i­ci­ans’ pro­duc­tivity. See my post In­nate Math­e­mat­i­cal Abil­ity. But I think that the per­cent of var­i­ance that in­tel­li­gence ex­plains is less than 50%.

• Ah, I see. I for­got about that, thanks!

the most effec­tive peo­ple in the world have a very spe­cific way of think­ing. They use their brain’s pat­tern-match­ing abil­ities to pro­cess the world, rather than us­ing ex­plicit reasoning

I don’t re­mem­ber that Glad­well gave any tips for ac­tu­ally de­vel­op­ing one’s skills for this type of think­ing, but he does have a num­ber of in­ter­est­ing sto­ries and anal­y­sis about this type of think­ing. It also makes the ob­ser­va­tion that this type of non-ex­plicit rea­son­ing can lead us astray.

I sus­pect that pat­tern-match­ing is vastly more effi­cient than ex­plicit rea­son­ing as you sug­gest, but that it is sub­ject to bias and can in some cases lead one astray. There­fore, I think that ra­tio­nal­ity is a com­bi­na­tion of both types of think­ing—to use your ex­am­ple, a math­e­mat­i­cian uses his/​her pat­tern match­ing thought pro­cesses to figure out how to ap­proach prov­ing a the­o­rem, but then rea­sons ex­plic­itly when for­mal­iz­ing the proof (and ex­plicit rea­son­ing would be used in ver­ify­ing the proof as well).

At least some of the bi­ases dis­cussed in the se­quences and el­se­where can be at­tributed to non-ex­plicit rea­son­ing and the an­ti­dote to these bi­ases is to rea­son ex­plic­itly.

• Thanks for your in­ter­est :-)

There’s cer­tainly over­lap, but I’m mak­ing a more pre­cise claim: that one can de­velop pow­er­ful in­tu­ition not only in par­tic­u­lar do­mains but that one can also de­velop pow­er­ful gen­eral pre­dic­tive mod­els to get very high epistemic ra­tio­nal­ity across the board.

I sus­pect that pat­tern-match­ing is vastly more effi­cient than ex­plicit rea­son­ing as you sug­gest, but that it is sub­ject to bias and can in some cases lead one astray.

Yes, my re­al­iza­tion is about rel­a­tive effect sizes: I used to think that the right bal­ance is 50% in­tu­ition and 50% ex­plicit rea­son­ing or some­thing, whereas now I think that it’s more like 95% in­tu­ition and 5% ex­plicit rea­son­ing. (I’m speak­ing very vaguely here.)

At least some of the bi­ases dis­cussed in the se­quences and el­se­where can be at­tributed to non-ex­plicit rea­son­ing and the an­ti­dote to these bi­ases is to rea­son ex­plic­itly.

Ah, but ex­plicit rea­son­ing isn’t the only an­ti­dote: you can also use in­tu­ition to cor­rect for emo­tional and cog­ni­tive bi­ases :-). I know that it’s highly nonob­vi­ous how one would go about do­ing this.

Some­what tan­gen­tially, you might be in­ter­ested by my post Rea­son is not the only means of over­com­ing bias. (The post is 4.5 years old..I’ve been think­ing about these things for a long time :P.)

• but I’m mak­ing a more pre­cise claim: that one can de­velop pow­er­ful in­tu­ition not only in par­tic­u­lar do­mains but that one can also de­velop pow­er­ful gen­eral pre­dic­tive mod­els to get very high epistemic ra­tio­nal­ity across the board.

Why do you think so? Ba­si­cally, what ev­i­dence do you have that you can build strong “in­tu­itions” which will work across di­verse do­mains? My off-the-top-of-my-head re­ac­tion is that in dis­similar do­mains your in­tu­ition will mis­lead you.

• It’s re­ally hard, that’s why al­most no­body knows how to do it :P.

Roughly speak­ing, the solu­tion for me was to de­velop deep in­tu­ition in a lot of differ­ent do­mains, ob­serve the fea­tures com­mon to the in­tu­itions in differ­ent do­mains, and ab­stract the com­mon fea­tures out.

Find­ing the com­mon fea­tures was very difficult, as there are a huge num­ber of con­found­ing fac­tors that mask over the un­der­ly­ing com­mon­al­ities. But it makes sense in hind­sight—we wouldn’t be able to de­velop deep in­tu­itions in so many differ­ent do­mains if not for there be­ing sub­tle un­der­ly­ing com­mon­al­ities—there weren’t evolu­tion­ary se­lec­tive pres­sures speci­fi­cally for the abil­ity to de­velop gen­eral rel­a­tivity and quan­tum field the­ory—the fact that it’s pos­si­ble for us means that the rele­vant pat­tern recog­ni­tion abil­ities are closely re­lated to the ones used in so­cial con­texts, etc.

• ob­serve the fea­tures com­mon to the in­tu­itions in differ­ent do­mains, and ab­stract the com­mon fea­tures out.

Have you ex­plic­itly fac­tored these out? If so, what are some ex­am­ples?

• It’s re­ally hard, that’s why al­most no­body knows how to do it :P.

The ques­tion is why do you think it is even pos­si­ble?

the solu­tion for me was to de­velop deep in­tu­ition in a lot of differ­ent do­mains, ob­serve the fea­tures com­mon to the in­tu­itions in differ­ent do­mains, and ab­stract the com­mon fea­tures out.

So, do you feel that your in­tu­ition will work suc­cess­fully in the fields of, say, post-mod­ernist liter­ary cri­tique, agri­cul­ture, and hu­man bio­chem­istry?

• The ques­tion is why do you think it is even pos­si­ble?

Be­cause I’ve seen other peo­ple do it, I’ve ob­served a strong cor­re­la­tion be­tween the abil­ity to do it and over­all func­tion­al­ity, and I’ve re­cently dis­cov­ered how to do it my­self and have seen huge gains to both my epistemic and in­stru­men­tal ra­tio­nal­ity.

I know that I’m not pro­vid­ing enough in­for­ma­tion for you to find what I’m say­ing very com­pel­ling. Again, it took me 10,000+ hours be­fore I my­self started to get it. I might well have been skep­ti­cal be­fore do­ing so.

So, do you feel that your in­tu­ition will work suc­cess­fully in the fields of, say, post-mod­ernist liter­ary cri­tique, agri­cul­ture, and hu­man bio­chem­istry?

I don’t know – it de­pends on the rel­a­tive roles of skill and luck in these fields. If you’re talk­ing about those ma­jor dis­cov­er­ies from the past that re­quired in­te­grat­ing a di­verse col­lec­tion of sources of in­for­ma­tion, I be­lieve that the peo­ple who made the dis­cov­er­ies were us­ing this style of think­ing. For ex­am­ple, I be­lieve that this was prob­a­bly true of Nor­man Bor­laug.

• Ah, but ex­plicit rea­son­ing isn’t the only antidote

Yes, I was just about to edit my post to say “an an­ti­dote” rather than “the an­ti­dote”. As a prac­ti­cal mat­ter, no one is go­ing to ex­plic­itly rea­son through ev­ery situ­a­tion. A more prac­ti­cal an­ti­dote is to rec­og­nize bi­ases and learn rules of thumb for avoid­ing them. A clas­sic ex­am­ple is the con­junc­tion fal­lacy. Ex­plic­itly calcu­lat­ing con­di­tional prob­a­bil­ities will ob­vi­ously cor­rect this fal­lacy, but most of us are not go­ing to do that most of the time. How­ever, if one is aware of the fal­lacy, one can de­velop a rule of thumb that states that less spe­cific hy­po­thet­i­cals are usu­ally more prob­a­ble than more spe­cific hy­po­thet­i­cals; this rule is suffi­cient for avoid­ing the con­junc­tion fal­lacy most of the time. How­ever, even here, ex­plicit rea­son­ing played a role in avoid­ing the bias; ex­plicit rea­son­ing was used to learn about and un­der­stand the bias, and to de­velop the rule of thumb.

Is us­ing this sort of rule of thumb what you mean by us­ing in­tu­ition to cor­rect for emo­tional and cog­ni­tive bi­ases?

• Oh this is nice. I’ve also come to re­al­ise this over time, ,in differ­ent words, and my mind is ex­tremely tick­led by how your for­mu­la­tion puts it on an equal foot­ing with other non-ex­plicit-ra­tio­nal­ity av­enues of thought.

I would love to help you. I am very in­ter­ested in a pas­sion pro­ject right now. And we seem to be clas­sify­ing similar things as hard-won re­al­i­sa­tions, though we have very differ­ent timelines for differ­ent things; talk­ing to you might be all-round in­ter­est­ing for me.

• Hi Jonah, this ar­ti­cle is very in­trigu­ing since I might be go­ing through a similar phase as you. Please add me to any list of col­lab­o­ra­tors you’re draw­ing up.

• Great :-). Send me an email at js­inick@gmail.com.

• This seems valuable—I’m in­ter­ested in helping (will email).

I want to high­light that “com­mu­ni­cat­ing how to do it” might not make sense as a frame. Pat­tern-match­ing is closely re­lated to chunk­ing. Ctrl+F yields other peo­ple who’ve men­tioned chess, so I’ll just point at that and then note that we ac­tu­ally know ex­actly how to com­mu­ni­cate the skill of chunk­ing chess­boards: you get the per­son to prac­tice chess in a cer­tain way. There are of course bet­ter and worse ways to do this, but it seems like rather than look­ing for an in­sight to com­mu­ni­cate you want to look for a learn­ing pro­cess and how to make it more effi­cient by (e.g.) tight­en­ing feed­back loops.

• I have a lot of ev­i­dence that this way of think­ing is how the most effec­tive peo­ple think about the world. Here I’ll give two ex­am­ples. Holden worked un­der Greg Jensen, the co-CEO of Bridge­wa­ter As­so­ci­ates, which is the largest hedge fund in the world.

BW also uses a lot of ex­plicit mod­els, https://​​www.youtube.com/​​watch?v=PHe0bXAIuk0

Holden work­ing un­der Greg is also gen­er­ally weak ev­i­dence about how Greg thinks.

• I wasn’t mak­ing an ar­gu­ment, I was stat­ing my po­si­tion. I have far more ev­i­dence than I can con­vey a sin­gle blog post.

• I per­son­ally agree with your core the­sis that pat­tern match­ing is cen­tral. I in­vested a lot of effort into Quan­tified Self com­mu­nity build­ing and gave press in­ter­views prais­ing the promise of QS. I think at the time I over­rated straight data over pat­tern match­ing. To­day I con­sider pat­tern match­ing much more im­por­tant. I’m happy to col­lab­o­rate on de­vel­op­ing this line of thought.

I’m weary of whether us­ing the word ‘ra­tio­nal­ity’ in this con­text is use­ful. Web­ster defines the word as: ‘the qual­ity or state of be­ing agree­able to rea­son’. Wikipe­dia says: ‘Ra­tion­al­ity is the qual­ity or state of be­ing rea­son­able, based on facts or rea­son. Ra­tion­al­ity im­plies the con­for­mity of one’s be­liefs with one’s rea­sons to be­lieve, or of one’s ac­tions with one’s rea­sons for ac­tion.’

• No, I haven’t. Thanks for point­ing it out. :-)

• The coolest pos­si­ble out­put of a col­lab­o­ra­tion like this would be some kind of browser-based game you could play that would level up your ra­tio­nal­ity.

Also, what char­ac­ter­is­tics/​skills does your ideal col­lab­o­ra­tor have? Maybe what you want to do is find an effec­tive al­tru­ist whose work could benefit very strongly from the skills you de­scribe, tu­tor them in the skills, and hav­ing taught 1 per­son, see if you can repli­cate the most effec­tive bits of teach­ing bits and scale them to a larger au­di­ence.

• This sounds like an ex­pla­na­tion for the old adage: “Go with your gut”. If your brain is a lot bet­ter at rec­og­niz­ing pat­terns than it is at draw­ing con­clu­sions through a chain of rea­son­ing, it seems ad­vis­able to trust that which your brain ex­cels at. Some­thing similar is brought up in The Gift of Fear, where the au­thor cites ex­am­ples where the pat­tern-recog­ni­tion sig­naled dan­ger, but peo­ple ig­nored them be­cause they could not come up with a chain of rea­son­ing to sup­port that con­clu­sion.

Suffi­ciently high qual­ity math­e­mat­i­ci­ans don’t make their dis­cov­er­ies through rea­son­ing. The math­e­mat­i­cal proof is the very last step: you do it to check that your eyes weren’t de­ceiv­ing you, but you know ahead of time that it’s your eyes prob­a­bly weren’t de­ceiv­ing you. Given that this is true even in math, which is thought of as the most log­i­cally rigor­ous sub­ject, it shouldn’t be sur­pris­ing that the same is true of epistemic ra­tio­nal­ity across the board.

In­ter­est­ing that you bring up this and Poin­care’s ex­pe­rience. Jac­ques Hadamard wrote a book ex­am­in­ing this phe­nomenon based on in­for­ma­tion he gath­ered from other math­e­mat­i­ci­ans as well as his (lay­man’s) knowl­edge of the psy­chol­ogy of the day. His con­clu­sions bore sev­eral similar­i­ties to what you’re try­ing to ex­plain in this post. He did, how­ever, note that ex­pe­riences like Poin­care’s gen­er­ally only took place if the re­searcher in ques­tion spent a lot of time work­ing on the prob­lem in the old “chain of rea­son­ing” way, with the pat­tern of­ten be­com­ing clear some weeks or months later, af­ter the re­searcher had moved on to a differ­ent prob­lem. Per­haps this is what con­sti­tutes train­ing one’s brain to see pat­terns.

• I read Hadamard’s book 8 years ago and liked it a lot.

What I missed is that I mis­tak­enly thought that Poin­care’s style of think­ing was re­served for su­per­ge­niuses, and that all that some­one like me could do was to clum­sily use ex­plicit rea­son­ing.

I found out oth­er­wise when I worked on my speed dat­ing pro­ject. Some­thing very pri­mal in me came out, and I worked on it al­most in­vol­un­tar­ily for ~90 hours a week for 12 weeks. I fi­nally had the ex­pe­rience of be­com­ing suffi­ciently deeply in­volved so that the prob­lems that I was try­ing to solve per­co­lated into my sub­con­scious and my in­tu­ition took over. I re­dis­cov­ered a large frac­tion of stan­dard ma­chine learn­ing al­gorithms (it was faster than learn­ing from books for me per­son­ally be­cause of my learn­ing dis­abil­ity). Be­fore this, I had no idea how ca­pa­ble I was. It made me re­al­ize that be­ing a great sci­en­tist might be within the reach of a much larger frac­tion of the pop­u­la­tion than I had thought.

• I don’t have a con­crete plan yet. I have draft posts that I’ve writ­ten that are in­suffi­ciently pol­ished for pub­li­ca­tion that I can share with you. You can get in touch with me at js­inick@gmail.com.

• What would be the goal of any such col­lab­o­ra­tion: LessWrong posts, a book, a pod­cast se­ries? Know­ing what you will pro­duce will help you sell your­self to po­ten­tial col­lab­o­ra­tors.

• I don’t know what the best route would be. Again, as far as I know, no­body has suc­ceeded in do­ing what I want to do, so there’s a prior against con­ven­tional ap­proachs work­ing. If I can in­ter­est Scott Alexan­der, I think that he might be able to do it, though if I re­call cor­rectly, he’s also blogged about how the more im­por­tant a topic is to his mind, the less views his posts on it get.