Argument, intuition, and recursion

Math­e­mat­i­ci­ans an­swer clean ques­tions that can be set­tled with for­mal ar­gu­ment. Scien­tists an­swer em­piri­cal ques­tions that can be set­tled with ex­per­i­men­ta­tion.

Col­lec­tive episte­mol­ogy is hard in do­mains where it’s hard to set­tle dis­putes with ei­ther for­mal ar­gu­ment or ex­per­i­men­ta­tion (or a com­bi­na­tion), like policy or fu­tur­ism.

I think that’s where ra­tio­nal­ists could add value, but first we have to grap­ple with a ba­sic ques­tion: if you can’t set­tle the ques­tion with logic, and you can’t check your in­tu­itions against re­al­ity to see how ac­cu­rate they are, then what are you even do­ing?

In this post I’ll ex­plain how I think about that ques­tion. For those who are pay­ing close at­ten­tion, it’s similar to one or two of my pre­vi­ous posts (e.g. 1 2 3 4 5...).

I. An example

An economist might an­swer a sim­ple ques­tion (“what is the ex­pected em­ploy­ment effect of a steel tar­iff?”) by set­ting up an econ 101 model and calcu­lat­ing equil­ibria.

After set­ting up enough sim­ple mod­els, they can de­velop in­tu­itions and heuris­tics that roughly pre­dict the out­come with­out ac­tu­ally do­ing the calcu­la­tion.

Th­ese in­tu­itions won’t be as ac­cu­rate as in­tu­itions trained against the real world—if our economist could ob­serve the im­pact of thou­sands of real eco­nomic in­ter­ven­tions, they should do that in­stead (and in the case of eco­nomics, you of­ten can). But the in­tu­ition isn’t vac­u­ous ei­ther: it’s a fast ap­prox­i­ma­tion of econ 101 mod­els.

Once our economist has built up econ 101 in­tu­itions, they can con­sider more nu­anced ar­gu­ments that lev­er­age those fast in­tu­itive judg­ments. For ex­am­ple, they could con­sider pos­si­ble mod­ifi­ca­tions to their sim­ple model of steel tar­iffs (like la­bor mar­ket fric­tions), use their in­tu­ition to quickly eval­u­ate each mod­ifi­ca­tion, and see which mod­ifi­ca­tions ac­tu­ally af­fect the sim­ple model’s con­clu­sion.

After go­ing through enough nu­anced ar­gu­ments, they can de­velop in­tu­itions and heuris­tics that pre­dict these out­comes. For ex­am­ple, they can learn to pre­dict which as­sump­tions are most im­por­tant to a sim­ple model’s con­clu­sions.

Equipped with these stronger in­tu­itions, our economist can use them to get bet­ter an­swers: they can con­struct more ro­bust mod­els, ex­plore the most im­por­tant as­sump­tions, de­sign more effec­tive ex­per­i­ments, and so on.

(Even­tu­ally our economist will im­prove their in­tu­itions fur­ther by pre­dict­ing these bet­ter an­swers; they can use the new in­tu­itions to an­swer more com­plex ques­tions....)

Any ques­tion that can be an­swered by this pro­ce­dure could even­tu­ally be an­swered us­ing econ 101 di­rectly. But with ev­ery iter­a­tion of in­tu­ition-build­ing, the com­plex­ity of the un­der­ly­ing econ 101 ex­pla­na­tion in­creases ge­o­met­ri­cally. This pro­cess won’t re­veal any truths be­yond those im­plicit in the econ 101 as­sump­tions, but it can do a good job of effi­ciently ex­plor­ing the log­i­cal con­se­quences of those as­sump­tions.

(In prac­tice, an economist’s in­tu­itions should in­cor­po­rate both the­o­ret­i­cal ar­gu­ment and rele­vant data, but that doesn’t change the ba­sic pic­ture.)

II. The process

The same re­cur­sive pro­cess is re­spon­si­ble for most of my in­tu­itions about fu­tur­ism. I don’t get to test my in­tu­ition by ac­tu­ally peek­ing at the world in 20 years. But I can con­sider ex­plicit ar­gu­ments and use them to re­fine my in­tu­itions—even if eval­u­at­ing ar­gu­ments re­quires us­ing my cur­rent in­tu­itions.

For ex­am­ple, when I think about take­off speeds I’m faced with ques­tions like “how much should we in­fer from the differ­ence be­tween chimps and hu­mans?” It’s not tractable to an­swer all of these sub­ques­tions in de­tail, so for a first pass I use my in­tu­ition to an­swer each sub­ques­tion.

Even­tu­ally it’s worth­while to ex­plore some of those sub­ques­tions in more depth, e.g. I might choose to ex­plore the anal­ogy be­tween chimps and hu­mans in more depth. In the pro­cess run into sub-sub-ques­tions, like “to what ex­tent is evolu­tion op­ti­miz­ing for the char­ac­ter­is­tics that changed dis­con­tin­u­ously be­tween chimps and hu­mans?” I ini­tially an­swer those sub­ques­tions with in­tu­ition but might some­times ex­pand them in the same way, turn­ing up sub-sub-sub-ques­tions...

When I ex­am­ine the ar­gu­ments for a ques­tion Q, I use my cur­rent in­tu­ition to an­swer the sub­ques­tions that I en­counter. Once I get an an­swer for Q, I do two things:

  • I up­date my cached be­lief about Q, to re­flect the new things I’ve learned.

  • If my new be­lief differs from my origi­nal in­tu­ition, I up­date my in­tu­ition. My in­tu­itions gen­er­al­ize across cases, so this will af­fect my view on lots of other ques­tions.

A naive de­scrip­tion of rea­son­ing only talks about the first kind of up­date. But I think that the sec­ond kind is where 99% of the im­por­tant stuff hap­pens.

(There isn’t any bright line be­tween these two cases. A “cached an­swer” is just a very spe­cific kind of in­tu­ition, and in prac­tice the ex­treme case of see­ing the ex­act ques­tion mul­ti­ple times is mostly ir­rele­vant. For ex­am­ple, it’s not helpful to have a cached an­swer to “how fast will AI take­off be?”; in­stead I have a cluster of in­tu­itions that gen­er­ate an­swers to a hun­dred differ­ent var­i­ants of that ques­tion.)

The sec­ond kind of up­date can come in lots of fla­vors. Some ex­am­ples:

  • When I make an in­tu­itive judg­ment I have to weigh lots of differ­ent fac­tors: my own snap judg­ment, oth­ers’ views, var­i­ous heuris­tic ar­gu­ments, var­i­ous analo­gies, etc. I set these weights partly based on em­piri­cal pre­dic­tions but largely based on pre­dict­ing the re­sult of ar­gu­ments. For ex­am­ple, in many con­texts I’d lean heav­ily on Carl or Holden’s views, based on them sys­tem­at­i­cally pre­dict­ing the views that I’d hold af­ter ex­plor­ing ar­gu­ments in more de­tail.

  • I have many ex­plicit heuris­tics or high-level prin­ci­ples of rea­son­ing that have been re­fined to pre­dict the re­sults of more de­tailed ar­gu­ments. For ex­am­ple, I of­ten use a cluster of “anti-fa­nat­i­cism” heuris­tics, against as­sign­ing un­bounded ra­tios be­tween the im­por­tance of differ­ent con­sid­er­a­tions. This is not ac­tu­ally a sim­ple gen­eral prin­ci­ple to state, and it’s not sup­ported by a gen­eral ar­gu­ment, in­stead I have an in­tu­itive sense of when the heuris­tic ap­plies.

  • My un­con­scious judg­ments are sig­nifi­cantly op­ti­mized to pre­dict the re­sult of longer ar­gu­ments. This is most ob­vi­ous in cases like math­e­mat­ics—for ex­am­ple, I have a well-de­vel­oped in­tu­itions about du­al­ity and the Fourier trans­form that lets me an­swer hard ques­tions, which was re­fined al­most en­tirely by prac­tice. In­tu­itions are harder to see (and less re­li­able) in cases like eco­nomics of foom or ro­bust­ness of RL to func­tion ap­prox­i­ma­tors, but some­thing ba­si­cally similar is go­ing on.

Note that none of these have in­de­pen­dent ev­i­den­tial value, they would be screened off by ex­plor­ing the ar­gu­ments in enough de­tail. But in prac­tice it’s pretty hard to do that, and in many cases might be com­pu­ta­tion­ally in­fea­si­ble.

Like the economist in the ex­am­ple, I would do bet­ter by up­dat­ing my in­tu­itions against the real world. But in many do­mains there just isn’t that much data—we only get to see one year of the fu­ture per year, and policy ex­per­i­ments can be very ex­pen­sive—and this ap­proach al­lows us to stretch the data we have by in­cor­po­rat­ing an in­creas­ing range of log­i­cal con­se­quences.

III. Disclaimer

The last sec­tion is partly a pos­i­tive de­scrip­tion of how I ac­tu­ally rea­son and partly a nor­ma­tive de­scrip­tion of how I be­lieve peo­ple should rea­son. In the next sec­tion I’ll try to turn it into a col­lec­tive episte­mol­ogy.

I’ve found this frame­work use­ful for clar­ify­ing my own think­ing about think­ing. Un­for­tu­nately, I can’t give you much em­piri­cal ev­i­dence that it works well.

Even if this ap­proach was the best thing since sliced bread, I think that em­piri­cally demon­strat­ing that it helps would still be a mas­sive sci­en­tific pro­ject. So I hope I can be for­given for a lack of em­piri­cal rigor. But you should still take ev­ery­thing with a grain of salt.

And I want to stress: I don’t mean to de­value div­ing deeply into ar­gu­ments and flesh­ing them out as much as pos­si­ble. I think it’s usu­ally im­pos­si­ble to get all the way to a math­e­mat­i­cal ar­gu­ment, but you can take a pretty gi­ant step from your ini­tial in­tu­itions. Though I talk about “one step back­ups” in the above ex­am­ples for sim­plic­ity, I think that up­dat­ing on re­ally big steps is of­ten a bet­ter idea. More­over, if we want to have the best view we can on a par­tic­u­lar ques­tion, it’s clearly worth un­pack­ing the ar­gu­ments as much as we can. (In fact the ar­gu­ment in this post should make you un­pack ar­gu­ments more, since in ad­di­tion to the ob­ject-level benefit you also benefit from build­ing stronger trans­ferrable in­tu­itions.)

IV. Disagreement

Sup­pose Alice and Bob dis­agree about a com­pli­cated ques­tion—say AI timelines—and they’d like to learn from each other.

A com­mon (im­plicit) hope is to ex­haus­tively ex­plore the tree of ar­gu­ments and coun­ter­ar­gu­ments, fol­low­ing a trail of higher-level dis­agree­ments to each low-level dis­agree­ment. If Alice and Bob mostly have similar in­tu­itions, but they’ve con­sid­ered differ­ent ar­gu­ments or have differ­ent em­piri­cal ev­i­dence, then this pro­cess can high­light the differ­ence and they can some­times reach agree­ment.

Often this doesn’t work be­cause Alice and Bob have wildly differ­ent in­tu­itions about a whole bunch of differ­ent ques­tions. I think that in a com­pli­cated ar­gu­ment, the num­ber of sub­ques­tions about which Alice and Bob can be as­tro­nom­i­cally large, and there is zero hope for re­solv­ing any sig­nifi­cant frac­tion of them. What to do then?

Here’s one pos­si­ble strat­egy. Let’s sup­pose for sim­plic­ity that Alice and Bob dis­agree, and that an out­side ob­server Judy is in­ter­ested in learn­ing about the truth of the mat­ter (the iden­ti­cal pro­ce­dure works if Judy is ac­tu­ally one of Alice and Bob). Then:

Alice ex­plains her view on the top level ques­tion, in terms of her an­swers to sim­pler sub­ques­tions. Bob likely dis­agrees with some of these steps. If there is dis­agree­ment, Alice and Bob talk un­til they “agree to dis­agree”—they make sure that they are us­ing the sub­ques­tion to mean the same thing, and that they’ve up­dated on each oth­ers’ be­liefs (and what­ever cur­sory ar­gu­ments each of them is will­ing to make about the claim). Then Alice and Bob find their most sig­nifi­cant dis­agree­ment and re­cur­sively ap­ply the same pro­cess to that dis­agree­ment.

They re­peat this pro­cess un­til they reach a state where they don’t have any sig­nifi­cant dis­agree­ments about sub­claims (po­ten­tially be­cause there are none, and the claim is so sim­ple that Judy feels con­fi­dent she can as­sess its truth di­rectly).

Hope­fully at this point Alice and Bob can reach agree­ment, or else iden­tify some im­plicit sub­ques­tion about which they dis­agree. But if not, that’s OK too. Ul­ti­mately Judy is the ar­biter of truth. Every time Alice and Bob have been dis­agree­ing, they have been mak­ing a claim about what Judy will ul­ti­mately be­lieve.

The rea­son we were ex­plor­ing this claim was be­cause Alice and Bob dis­agreed sig­nifi­cantly be­fore we un­packed the de­tails. Now at least one of Alice and Bob learns that they were wrong, and both of them can up­date their in­tu­itions (in­clud­ing their in­tu­itions for how much to re­spect each oth­ers’ opinions in differ­ent kinds of cases).

Alice and Bob then start the pro­cess over with their new in­tu­itions. The new pro­cess might in­volve pur­su­ing a nearly-iden­ti­cal set of dis­agree­ments (which they can do ex­tremely quickly), but at some point it will take a differ­ent turn.

If you run this pro­cess enough times, even­tu­ally (at least one of) Alice or Bob will change their opinion about the root ques­tion—or more pre­cisely, about what Judy will even­tu­ally come to be­lieve about the root ques­tion—be­cause they’ve ab­sorbed some­thing about the oth­ers’ in­tu­ition.

There are two qual­i­ta­tively differ­ent ways that agree­ment can oc­cur:

  • Con­ver­gence. Even­tu­ally, Alice will have ab­sorbed Bob’s in­tu­itions and vice versa. This might take a while—po­ten­tially, as long as it took Alice or Bob to origi­nally de­velop their in­tu­itions. (But it can still be ex­po­nen­tially smaller than the size of the tree.)

  • Mu­tual re­spect. If Alice and Bob keep dis­agree­ing sig­nifi­cantly, then the sim­ple al­gorithm “take the av­er­age of Alice and Bob’s view” will out­perform at least one of them (and of­ten both of them). So two Bayesi­ans can’t dis­agree sig­nifi­cantly too many times, even if they to­tally dis­trust one an­other.

If Alice and Bob are poor Bayesi­ans (or mo­ti­vated rea­son­ers) and con­tinue to dis­agree, then Judy can eas­ily take the mat­ter into her own hands by de­cid­ing how to weigh Alice and Bob’s opinions. For ex­am­ple, Judy might de­cide that Alice is right most of the time and Bob is be­ing silly by not defer­ring more—or Judy might de­cide that both of them are silly and that the mid­point be­tween their views is even bet­ter.

The key thing that makes this work—and the rea­son it re­quires no com­mon knowl­edge of ra­tio­nal­ity or other strong as­sump­tions—is that Alice and Bob can cash out their dis­agree­ments as a pre­dic­tion about what Judy will ul­ti­mately be­lieve.

Although it in­tro­duces sig­nifi­cant ad­di­tional com­pli­ca­tions, I think this en­tire scheme would some­times work bet­ter with bet­ting, as in this pro­posal. Rather than trust­ing Alice and Bob to be rea­son­able Bayesi­ans and even­tu­ally stop dis­agree­ing sig­nifi­cantly, Judy can in­stead perform an ex­plicit ar­bi­trage be­tween their views. This only works if Alice and Bob both care about Judy’s view and are will­ing to pay to in­fluence it.

V. As­sorted details

After con­ver­gence Alice and Bob agree only ap­prox­i­mately about each claim (such that they won’t up­date much from re­solv­ing the dis­agree­ment). Hope­fully that lets them agree ap­prox­i­mately about the top-level claim. If sub­tle dis­agree­ments about lem­mas can blow up to gi­ant dis­agree­ments about down­stream claims, then this pro­cess won’t gen­er­ally con­verge. If Alice and Bob are care­ful prob­a­bil­is­tic rea­son­ers, then a “slight” dis­agree­ment in­volves each of them ac­knowl­edg­ing the plau­si­bil­ity of the oth­ers’ view, which seems to rule out most kinds of cas­cad­ing dis­agree­ment.

This is not nec­es­sar­ily an effec­tive tool for Alice to blud­geon Judy into adopt­ing her view, it’s only helpful if Judy is ac­tu­ally try­ing to learn some­thing. If you are try­ing to blud­geon peo­ple with ar­gu­ments, you are prob­a­bly do­ing it wrong. (Though gosh there are a lot of ex­am­ples of this amongst the ra­tio­nal­ists.)

By the con­struc­tion of the pro­ce­dure, Alice and Bob are hav­ing dis­agree­ments about what Judy will be­lieve af­ter ex­am­in­ing ar­gu­ments. This pro­ce­dure is (at best) go­ing to ex­tract the log­i­cal con­se­quences of Judy’s be­liefs and stan­dards of ev­i­dence.

Alice and Bob don’t have to op­er­a­tional­ize claims enough that they can bet on them. But they do want to reach agree­ment about the mean­ing of each sub­ques­tion, and in par­tic­u­lar un­der­stand what mean­ing Judy as­signs to each sub­ques­tion. “Mean­ing” cap­tures both what you in­fer from an an­swer to that sub­ques­tion, and how you an­swer it). If Alice and Bob don’t know how Judy uses lan­guage, then they can learn that over the course of this pro­cess, but hope­fully we have more cost-effec­tive ways to agree on the use of lan­guage (or com­mu­ni­cate on­tolo­gies) than go­ing through an elab­o­rate ar­gu­ment pro­ce­dure.

One way that Alice and Bob can get stuck is by not trust­ing each oth­ers’ em­piri­cal ev­i­dence. For ex­am­ple, Bob might ex­plain his be­liefs by say­ing that he’s seen ev­i­dence X, and Alice might not trust him or might be­lieve that he is re­port­ing ev­i­dence se­lec­tively. This pro­ce­dure isn’t go­ing to re­solve that kind of dis­agree­ment. Ul­ti­mately it just punts the ques­tion to what Judy is will­ing to be­lieve based on all of the available ar­gu­ments.

Alice and Bob’s ar­gu­ment can have loops, if e.g. Alice be­lieve X be­cause of Y, which she be­lieves be­cause of X. We can un­wind these loops by tag­ging an­swers ex­plic­itly with the “depth” of rea­son­ing sup­port­ing that an­swer, decre­ment­ing the depth at each step, and de­fault­ing to Judy’s in­tu­ition when the depth reaches 0. This mir­rors the iter­a­tive pro­cess of in­tu­ition-for­ma­tion which evolves over time, start­ing from t=0 when we use our ini­tial in­tu­itions. I think that in prac­tice this is usu­ally not needed in ar­gu­ments, be­cause ev­ery­one knows why Alice is try­ing to ar­gue for X—if Alice is try­ing to prove X as a step to­wards prov­ing Y, then in­vok­ing Y as a lemma for prov­ing X looks weak.

My fu­tur­ism ex­am­ples differ from my economist ex­am­ple in that I’m start­ing from big ques­tions, and break­ing them down to figure out what low-level ques­tions are im­por­tant, rather than start­ing from a set of tech­niques and com­pos­ing them to see what big­ger-pic­ture ques­tions I can an­swer. In prac­tice I think that both tech­niques are ap­pro­pri­ate and a com­bi­na­tion usu­ally makes the most sense. In the con­text of ar­gu­ment in par­tic­u­lar, I think that break­ing down is a par­tic­u­larly valuable strat­egy. But even in ar­gu­ments it’s still of­ten faster go on an in­tu­ition-build­ing di­gres­sion where we con­sider sub­ques­tions that haven’t ap­peared ex­plic­itly in the ar­gu­ment.