Argument, intuition, and recursion

Math­e­mat­i­ci­ans an­swer clean ques­tions that can be set­tled with for­mal ar­gu­ment. Scien­tists an­swer em­piri­cal ques­tions that can be set­tled with ex­per­i­men­ta­tion.

Col­lec­tive episte­mol­ogy is hard in do­mains where it’s hard to set­tle dis­putes with ei­ther for­mal ar­gu­ment or ex­per­i­men­ta­tion (or a com­bi­na­tion), like policy or fu­tur­ism.

I think that’s where ra­tio­nal­ists could add value, but first we have to grap­ple with a ba­sic ques­tion: if you can’t set­tle the ques­tion with logic, and you can’t check your in­tu­itions against re­al­ity to see how ac­cu­rate they are, then what are you even do­ing?

In this post I’ll ex­plain how I think about that ques­tion. For those who are pay­ing close at­ten­tion, it’s similar to one or two of my pre­vi­ous posts (e.g. 1 2 3 4 5...).

I. An example

An economist might an­swer a sim­ple ques­tion (“what is the ex­pected em­ploy­ment effect of a steel tar­iff?”) by set­ting up an econ 101 model and calcu­lat­ing equil­ibria.

After set­ting up enough sim­ple mod­els, they can de­velop in­tu­itions and heuris­tics that roughly pre­dict the out­come with­out ac­tu­ally do­ing the calcu­la­tion.

Th­ese in­tu­itions won’t be as ac­cu­rate as in­tu­itions trained against the real world—if our economist could ob­serve the im­pact of thou­sands of real eco­nomic in­ter­ven­tions, they should do that in­stead (and in the case of eco­nomics, you of­ten can). But the in­tu­ition isn’t vac­u­ous ei­ther: it’s a fast ap­prox­i­ma­tion of econ 101 mod­els.

Once our economist has built up econ 101 in­tu­itions, they can con­sider more nu­anced ar­gu­ments that lev­er­age those fast in­tu­itive judg­ments. For ex­am­ple, they could con­sider pos­si­ble mod­ifi­ca­tions to their sim­ple model of steel tar­iffs (like la­bor mar­ket fric­tions), use their in­tu­ition to quickly eval­u­ate each mod­ifi­ca­tion, and see which mod­ifi­ca­tions ac­tu­ally af­fect the sim­ple model’s con­clu­sion.

After go­ing through enough nu­anced ar­gu­ments, they can de­velop in­tu­itions and heuris­tics that pre­dict these out­comes. For ex­am­ple, they can learn to pre­dict which as­sump­tions are most im­por­tant to a sim­ple model’s con­clu­sions.

Equipped with these stronger in­tu­itions, our economist can use them to get bet­ter an­swers: they can con­struct more ro­bust mod­els, ex­plore the most im­por­tant as­sump­tions, de­sign more effec­tive ex­per­i­ments, and so on.

(Even­tu­ally our economist will im­prove their in­tu­itions fur­ther by pre­dict­ing these bet­ter an­swers; they can use the new in­tu­itions to an­swer more com­plex ques­tions....)

Any ques­tion that can be an­swered by this pro­ce­dure could even­tu­ally be an­swered us­ing econ 101 di­rectly. But with ev­ery iter­a­tion of in­tu­ition-build­ing, the com­plex­ity of the un­der­ly­ing econ 101 ex­pla­na­tion in­creases ge­o­met­ri­cally. This pro­cess won’t re­veal any truths be­yond those im­plicit in the econ 101 as­sump­tions, but it can do a good job of effi­ciently ex­plor­ing the log­i­cal con­se­quences of those as­sump­tions.

(In prac­tice, an economist’s in­tu­itions should in­cor­po­rate both the­o­ret­i­cal ar­gu­ment and rele­vant data, but that doesn’t change the ba­sic pic­ture.)

II. The process

The same re­cur­sive pro­cess is re­spon­si­ble for most of my in­tu­itions about fu­tur­ism. I don’t get to test my in­tu­ition by ac­tu­ally peek­ing at the world in 20 years. But I can con­sider ex­plicit ar­gu­ments and use them to re­fine my in­tu­itions—even if eval­u­at­ing ar­gu­ments re­quires us­ing my cur­rent in­tu­itions.

For ex­am­ple, when I think about take­off speeds I’m faced with ques­tions like “how much should we in­fer from the differ­ence be­tween chimps and hu­mans?” It’s not tractable to an­swer all of these sub­ques­tions in de­tail, so for a first pass I use my in­tu­ition to an­swer each sub­ques­tion.

Even­tu­ally it’s worth­while to ex­plore some of those sub­ques­tions in more depth, e.g. I might choose to ex­plore the anal­ogy be­tween chimps and hu­mans in more depth. In the pro­cess run into sub-sub-ques­tions, like “to what ex­tent is evolu­tion op­ti­miz­ing for the char­ac­ter­is­tics that changed dis­con­tin­u­ously be­tween chimps and hu­mans?” I ini­tially an­swer those sub­ques­tions with in­tu­ition but might some­times ex­pand them in the same way, turn­ing up sub-sub-sub-ques­tions...

When I ex­am­ine the ar­gu­ments for a ques­tion Q, I use my cur­rent in­tu­ition to an­swer the sub­ques­tions that I en­counter. Once I get an an­swer for Q, I do two things:

• I up­date my cached be­lief about Q, to re­flect the new things I’ve learned.

• If my new be­lief differs from my origi­nal in­tu­ition, I up­date my in­tu­ition. My in­tu­itions gen­er­al­ize across cases, so this will af­fect my view on lots of other ques­tions.

A naive de­scrip­tion of rea­son­ing only talks about the first kind of up­date. But I think that the sec­ond kind is where 99% of the im­por­tant stuff hap­pens.

(There isn’t any bright line be­tween these two cases. A “cached an­swer” is just a very spe­cific kind of in­tu­ition, and in prac­tice the ex­treme case of see­ing the ex­act ques­tion mul­ti­ple times is mostly ir­rele­vant. For ex­am­ple, it’s not helpful to have a cached an­swer to “how fast will AI take­off be?”; in­stead I have a cluster of in­tu­itions that gen­er­ate an­swers to a hun­dred differ­ent var­i­ants of that ques­tion.)

The sec­ond kind of up­date can come in lots of fla­vors. Some ex­am­ples:

• When I make an in­tu­itive judg­ment I have to weigh lots of differ­ent fac­tors: my own snap judg­ment, oth­ers’ views, var­i­ous heuris­tic ar­gu­ments, var­i­ous analo­gies, etc. I set these weights partly based on em­piri­cal pre­dic­tions but largely based on pre­dict­ing the re­sult of ar­gu­ments. For ex­am­ple, in many con­texts I’d lean heav­ily on Carl or Holden’s views, based on them sys­tem­at­i­cally pre­dict­ing the views that I’d hold af­ter ex­plor­ing ar­gu­ments in more de­tail.

• I have many ex­plicit heuris­tics or high-level prin­ci­ples of rea­son­ing that have been re­fined to pre­dict the re­sults of more de­tailed ar­gu­ments. For ex­am­ple, I of­ten use a cluster of “anti-fa­nat­i­cism” heuris­tics, against as­sign­ing un­bounded ra­tios be­tween the im­por­tance of differ­ent con­sid­er­a­tions. This is not ac­tu­ally a sim­ple gen­eral prin­ci­ple to state, and it’s not sup­ported by a gen­eral ar­gu­ment, in­stead I have an in­tu­itive sense of when the heuris­tic ap­plies.

• My un­con­scious judg­ments are sig­nifi­cantly op­ti­mized to pre­dict the re­sult of longer ar­gu­ments. This is most ob­vi­ous in cases like math­e­mat­ics—for ex­am­ple, I have a well-de­vel­oped in­tu­itions about du­al­ity and the Fourier trans­form that lets me an­swer hard ques­tions, which was re­fined al­most en­tirely by prac­tice. In­tu­itions are harder to see (and less re­li­able) in cases like eco­nomics of foom or ro­bust­ness of RL to func­tion ap­prox­i­ma­tors, but some­thing ba­si­cally similar is go­ing on.

Note that none of these have in­de­pen­dent ev­i­den­tial value, they would be screened off by ex­plor­ing the ar­gu­ments in enough de­tail. But in prac­tice it’s pretty hard to do that, and in many cases might be com­pu­ta­tion­ally in­fea­si­ble.

Like the economist in the ex­am­ple, I would do bet­ter by up­dat­ing my in­tu­itions against the real world. But in many do­mains there just isn’t that much data—we only get to see one year of the fu­ture per year, and policy ex­per­i­ments can be very ex­pen­sive—and this ap­proach al­lows us to stretch the data we have by in­cor­po­rat­ing an in­creas­ing range of log­i­cal con­se­quences.

III. Disclaimer

The last sec­tion is partly a pos­i­tive de­scrip­tion of how I ac­tu­ally rea­son and partly a nor­ma­tive de­scrip­tion of how I be­lieve peo­ple should rea­son. In the next sec­tion I’ll try to turn it into a col­lec­tive episte­mol­ogy.

I’ve found this frame­work use­ful for clar­ify­ing my own think­ing about think­ing. Un­for­tu­nately, I can’t give you much em­piri­cal ev­i­dence that it works well.

Even if this ap­proach was the best thing since sliced bread, I think that em­piri­cally demon­strat­ing that it helps would still be a mas­sive sci­en­tific pro­ject. So I hope I can be for­given for a lack of em­piri­cal rigor. But you should still take ev­ery­thing with a grain of salt.

And I want to stress: I don’t mean to de­value div­ing deeply into ar­gu­ments and flesh­ing them out as much as pos­si­ble. I think it’s usu­ally im­pos­si­ble to get all the way to a math­e­mat­i­cal ar­gu­ment, but you can take a pretty gi­ant step from your ini­tial in­tu­itions. Though I talk about “one step back­ups” in the above ex­am­ples for sim­plic­ity, I think that up­dat­ing on re­ally big steps is of­ten a bet­ter idea. More­over, if we want to have the best view we can on a par­tic­u­lar ques­tion, it’s clearly worth un­pack­ing the ar­gu­ments as much as we can. (In fact the ar­gu­ment in this post should make you un­pack ar­gu­ments more, since in ad­di­tion to the ob­ject-level benefit you also benefit from build­ing stronger trans­ferrable in­tu­itions.)

IV. Disagreement

Sup­pose Alice and Bob dis­agree about a com­pli­cated ques­tion—say AI timelines—and they’d like to learn from each other.

A com­mon (im­plicit) hope is to ex­haus­tively ex­plore the tree of ar­gu­ments and coun­ter­ar­gu­ments, fol­low­ing a trail of higher-level dis­agree­ments to each low-level dis­agree­ment. If Alice and Bob mostly have similar in­tu­itions, but they’ve con­sid­ered differ­ent ar­gu­ments or have differ­ent em­piri­cal ev­i­dence, then this pro­cess can high­light the differ­ence and they can some­times reach agree­ment.

Often this doesn’t work be­cause Alice and Bob have wildly differ­ent in­tu­itions about a whole bunch of differ­ent ques­tions. I think that in a com­pli­cated ar­gu­ment, the num­ber of sub­ques­tions about which Alice and Bob can be as­tro­nom­i­cally large, and there is zero hope for re­solv­ing any sig­nifi­cant frac­tion of them. What to do then?

Here’s one pos­si­ble strat­egy. Let’s sup­pose for sim­plic­ity that Alice and Bob dis­agree, and that an out­side ob­server Judy is in­ter­ested in learn­ing about the truth of the mat­ter (the iden­ti­cal pro­ce­dure works if Judy is ac­tu­ally one of Alice and Bob). Then:

Alice ex­plains her view on the top level ques­tion, in terms of her an­swers to sim­pler sub­ques­tions. Bob likely dis­agrees with some of these steps. If there is dis­agree­ment, Alice and Bob talk un­til they “agree to dis­agree”—they make sure that they are us­ing the sub­ques­tion to mean the same thing, and that they’ve up­dated on each oth­ers’ be­liefs (and what­ever cur­sory ar­gu­ments each of them is will­ing to make about the claim). Then Alice and Bob find their most sig­nifi­cant dis­agree­ment and re­cur­sively ap­ply the same pro­cess to that dis­agree­ment.

They re­peat this pro­cess un­til they reach a state where they don’t have any sig­nifi­cant dis­agree­ments about sub­claims (po­ten­tially be­cause there are none, and the claim is so sim­ple that Judy feels con­fi­dent she can as­sess its truth di­rectly).

Hope­fully at this point Alice and Bob can reach agree­ment, or else iden­tify some im­plicit sub­ques­tion about which they dis­agree. But if not, that’s OK too. Ul­ti­mately Judy is the ar­biter of truth. Every time Alice and Bob have been dis­agree­ing, they have been mak­ing a claim about what Judy will ul­ti­mately be­lieve.

The rea­son we were ex­plor­ing this claim was be­cause Alice and Bob dis­agreed sig­nifi­cantly be­fore we un­packed the de­tails. Now at least one of Alice and Bob learns that they were wrong, and both of them can up­date their in­tu­itions (in­clud­ing their in­tu­itions for how much to re­spect each oth­ers’ opinions in differ­ent kinds of cases).

Alice and Bob then start the pro­cess over with their new in­tu­itions. The new pro­cess might in­volve pur­su­ing a nearly-iden­ti­cal set of dis­agree­ments (which they can do ex­tremely quickly), but at some point it will take a differ­ent turn.

If you run this pro­cess enough times, even­tu­ally (at least one of) Alice or Bob will change their opinion about the root ques­tion—or more pre­cisely, about what Judy will even­tu­ally come to be­lieve about the root ques­tion—be­cause they’ve ab­sorbed some­thing about the oth­ers’ in­tu­ition.

There are two qual­i­ta­tively differ­ent ways that agree­ment can oc­cur:

• Con­ver­gence. Even­tu­ally, Alice will have ab­sorbed Bob’s in­tu­itions and vice versa. This might take a while—po­ten­tially, as long as it took Alice or Bob to origi­nally de­velop their in­tu­itions. (But it can still be ex­po­nen­tially smaller than the size of the tree.)

• Mu­tual re­spect. If Alice and Bob keep dis­agree­ing sig­nifi­cantly, then the sim­ple al­gorithm “take the av­er­age of Alice and Bob’s view” will out­perform at least one of them (and of­ten both of them). So two Bayesi­ans can’t dis­agree sig­nifi­cantly too many times, even if they to­tally dis­trust one an­other.

If Alice and Bob are poor Bayesi­ans (or mo­ti­vated rea­son­ers) and con­tinue to dis­agree, then Judy can eas­ily take the mat­ter into her own hands by de­cid­ing how to weigh Alice and Bob’s opinions. For ex­am­ple, Judy might de­cide that Alice is right most of the time and Bob is be­ing silly by not defer­ring more—or Judy might de­cide that both of them are silly and that the mid­point be­tween their views is even bet­ter.

The key thing that makes this work—and the rea­son it re­quires no com­mon knowl­edge of ra­tio­nal­ity or other strong as­sump­tions—is that Alice and Bob can cash out their dis­agree­ments as a pre­dic­tion about what Judy will ul­ti­mately be­lieve.

Although it in­tro­duces sig­nifi­cant ad­di­tional com­pli­ca­tions, I think this en­tire scheme would some­times work bet­ter with bet­ting, as in this pro­posal. Rather than trust­ing Alice and Bob to be rea­son­able Bayesi­ans and even­tu­ally stop dis­agree­ing sig­nifi­cantly, Judy can in­stead perform an ex­plicit ar­bi­trage be­tween their views. This only works if Alice and Bob both care about Judy’s view and are will­ing to pay to in­fluence it.

V. As­sorted details

After con­ver­gence Alice and Bob agree only ap­prox­i­mately about each claim (such that they won’t up­date much from re­solv­ing the dis­agree­ment). Hope­fully that lets them agree ap­prox­i­mately about the top-level claim. If sub­tle dis­agree­ments about lem­mas can blow up to gi­ant dis­agree­ments about down­stream claims, then this pro­cess won’t gen­er­ally con­verge. If Alice and Bob are care­ful prob­a­bil­is­tic rea­son­ers, then a “slight” dis­agree­ment in­volves each of them ac­knowl­edg­ing the plau­si­bil­ity of the oth­ers’ view, which seems to rule out most kinds of cas­cad­ing dis­agree­ment.

This is not nec­es­sar­ily an effec­tive tool for Alice to blud­geon Judy into adopt­ing her view, it’s only helpful if Judy is ac­tu­ally try­ing to learn some­thing. If you are try­ing to blud­geon peo­ple with ar­gu­ments, you are prob­a­bly do­ing it wrong. (Though gosh there are a lot of ex­am­ples of this amongst the ra­tio­nal­ists.)

By the con­struc­tion of the pro­ce­dure, Alice and Bob are hav­ing dis­agree­ments about what Judy will be­lieve af­ter ex­am­in­ing ar­gu­ments. This pro­ce­dure is (at best) go­ing to ex­tract the log­i­cal con­se­quences of Judy’s be­liefs and stan­dards of ev­i­dence.

Alice and Bob don’t have to op­er­a­tional­ize claims enough that they can bet on them. But they do want to reach agree­ment about the mean­ing of each sub­ques­tion, and in par­tic­u­lar un­der­stand what mean­ing Judy as­signs to each sub­ques­tion. “Mean­ing” cap­tures both what you in­fer from an an­swer to that sub­ques­tion, and how you an­swer it). If Alice and Bob don’t know how Judy uses lan­guage, then they can learn that over the course of this pro­cess, but hope­fully we have more cost-effec­tive ways to agree on the use of lan­guage (or com­mu­ni­cate on­tolo­gies) than go­ing through an elab­o­rate ar­gu­ment pro­ce­dure.

One way that Alice and Bob can get stuck is by not trust­ing each oth­ers’ em­piri­cal ev­i­dence. For ex­am­ple, Bob might ex­plain his be­liefs by say­ing that he’s seen ev­i­dence X, and Alice might not trust him or might be­lieve that he is re­port­ing ev­i­dence se­lec­tively. This pro­ce­dure isn’t go­ing to re­solve that kind of dis­agree­ment. Ul­ti­mately it just punts the ques­tion to what Judy is will­ing to be­lieve based on all of the available ar­gu­ments.

Alice and Bob’s ar­gu­ment can have loops, if e.g. Alice be­lieve X be­cause of Y, which she be­lieves be­cause of X. We can un­wind these loops by tag­ging an­swers ex­plic­itly with the “depth” of rea­son­ing sup­port­ing that an­swer, decre­ment­ing the depth at each step, and de­fault­ing to Judy’s in­tu­ition when the depth reaches 0. This mir­rors the iter­a­tive pro­cess of in­tu­ition-for­ma­tion which evolves over time, start­ing from t=0 when we use our ini­tial in­tu­itions. I think that in prac­tice this is usu­ally not needed in ar­gu­ments, be­cause ev­ery­one knows why Alice is try­ing to ar­gue for X—if Alice is try­ing to prove X as a step to­wards prov­ing Y, then in­vok­ing Y as a lemma for prov­ing X looks weak.

My fu­tur­ism ex­am­ples differ from my economist ex­am­ple in that I’m start­ing from big ques­tions, and break­ing them down to figure out what low-level ques­tions are im­por­tant, rather than start­ing from a set of tech­niques and com­pos­ing them to see what big­ger-pic­ture ques­tions I can an­swer. In prac­tice I think that both tech­niques are ap­pro­pri­ate and a com­bi­na­tion usu­ally makes the most sense. In the con­text of ar­gu­ment in par­tic­u­lar, I think that break­ing down is a par­tic­u­larly valuable strat­egy. But even in ar­gu­ments it’s still of­ten faster go on an in­tu­ition-build­ing di­gres­sion where we con­sider sub­ques­tions that haven’t ap­peared ex­plic­itly in the ar­gu­ment.

• As I said at the time

The ques­tion of how to im­prove our in­tu­itions around top­ics with few and slow feed­back loops is a cen­tral ques­tion of ra­tio­nal­ity, and this post added a lot of helpful ex­plicit mod­els to this prob­lem that I’ve not seen put quite this way any­where else.

I con­tinue to think about some of the ideas in this post reg­u­larly.

• I’ve got­ten a lot of value out of posts in the refer­ence class of “at­tempts at some­what com­plete mod­els of what good rea­son­ing looks like”, and this one has been one of them.

I don’t think I fully agree with the model out­lined here, but I think the post did suc­ceed at adding it to my toolbox.

• This is a nice, sim­ple model for think­ing. But I no­tice that both logic and em­piri­cism some­times have “short­cuts” — non-ob­vi­ous ways to shorten, or oth­er­wise sub­stan­tially ro­bus­tify, the chain of (logic/​ev­i­dence). It’s rea­son­able to imag­ine that in­tu­ition/​ra­tio­nal­ity would also have var­i­ous short­cuts; some that would cor­re­spond to log­i­cal/​em­piri­cal short­cuts, and some that would be differ­ent. Com­mu­ni­ca­tion is more difficult when two peo­ple are us­ing chains of rea­son­ing that differ sub­stan­tially in what short­cuts they use. You could get two valid ar­gu­ments on a ques­tion, and be able to rec­og­nize the val­idity of each, but be al­most com­pletely at a loss when try­ing to com­bine those two into an over­all judge­ment.

Oops, I guess that was more of a com­ment than a re­view. At re­view-level, what I meant to say was: nice foun­da­tion, but it’s clear this doesn’t ex­haust the ques­tion. Which is good.

• I like the economist ex­am­ple as a good illus­tra­tion of the pro­cess, but it also makes me slightly re­duce my (already-low) con­fi­dence in our abil­ity to make our sen­si­ble pre­dic­tions about things like take-off speeds, given that economists say things like this:

As my col­league Jeffrey Fried­man ar­gues, ex­pert pre­dic­tions about the the likely effects of chang­ing a sin­gle policy tend to be pretty bad. I’ll use my­self as an ex­am­ple. I’ve fol­lowed the aca­demic liter­a­ture about the min­i­mum wage for al­most twenty years, and I’m an ex­pe­rienced, pro­fes­sional policy an­a­lyst, so I’ve got a weak claim to ex­per­tise in the sub­ject. What do I have to show for that? Not much, re­ally. I’ve got strong in­tu­itions about the likely effects of rais­ing min­i­mum wages in var­i­ous con­texts. But all I re­ally know is that the con­text mat­ters a great deal, that a lot of in­ter­re­lated fac­tors af­fect the dy­nam­ics of low-wage la­bor mar­kets, and that I can’t say in ad­vance which mar­gin will ad­just when the wage floor is raised. In­deed, whether we should ex­pect in­creases in the min­i­mum wage to hurt or help low-wage work­ers is a ques­tion No­bel Prize-win­ning economists dis­agree about. La­bor mar­kets are com­pli­cated!

Which in my mind could be sum­ma­rized as “af­ter 20 years of study­ing the the­ory and prac­ti­cal stud­ies of this topic, I’ve got strong in­tu­itions, but in prac­tice they aren’t enough for me to make strong pre­dic­tions”. And it seems that the ques­tion of min­i­mum wage is one for which there is much more di­rect ev­i­dence than there is for some­thing like take-off speeds, sug­gest­ing that we should be able even less able to make good pre­dic­tions about that.

• The usual im­plicit ap­proach is to ex­plore the tree of ar­gu­ments and coun­ter­ar­gu­ments, us­ing dis­agree­ment as a heuris­tic to pri­ori­tize which points to ex­plore in more de­tail… Often this doesn’t work be­cause Alice and Bob have wildly differ­ent in­tu­itions about a whole bunch of differ­ent ques­tions… What to do then?

Apolo­gies for be­ing dense, but I would also de­scribe the pro­ce­dure that you lay out fol­low­ing this bit as a “tree of ar­gu­ments and coun­ter­ar­gu­ments, us­ing dis­agree­ment as a heuris­tic to pri­ori­tize which points to ex­plore”.

What’s the cru­cial differ­ence be­tween the pro­ce­dure you de­scribe and the usual ap­proach? Is it that when­ever you hit on any sub­ques­tion sim­ple enough to re­solve, ev­ery­one up­dates their in­tu­itions and you start again from the top?

(“They re­peat this pro­cess un­til they reach a state where they don’t have any sig­nifi­cant dis­agree­ments about sub­claims… Alice and Bob then start the pro­cess over with their new in­tu­itions.”)

• Changed to: “A com­mon (im­plicit) hope is to ex­haus­tively ex­plore the tree of ar­gu­ments and coun­ter­ar­gu­ments, fol­low­ing a trail of higher-level dis­agree­ments to each low-level dis­agree­ment.”

I’m dis­t­in­guish­ing be­tween hop­ing to change your view about the root ques­tion by get­ting to each dis­agree­ment in turn and prop­a­gat­ing the log­i­cal con­se­quences of re­solv­ing it, or view­ing each dis­agree­ment as an ob­ser­va­tion that can help re­fine in­tu­ition /​ ar­bi­trate be­tween con­flict­ing in­tu­itions.

• I cu­rated this post for the fol­low­ing rea­sons:

• The ques­tion of how to im­prove our in­tu­itions around top­ics with few and slow feed­back loops is a cen­tral ques­tion of ra­tio­nal­ity, and this post added a lot of helpful ex­plicit mod­els to this prob­lem that I’ve not seen put quite this way any­where else.

• The core ideas seem not only valuable epistem­i­cally, but also to un­derly some promis­ing al­ign­ment strate­gies I’ve seen (that I be­lieve in­herit from you, though I’m un­cer­tain about this point).

The biggest hes­i­ta­tion I had with cu­rat­ing this post:

• Each sec­tion is very de­tailed, and it took me a sur­pris­ing amount of work to un­der­stand the struc­ture of and suc­cess­ful chunk both the post over­all and its sub­sec­tions. Espe­cially so given its length.

Over­all I am ex­cited to sit with these mod­els for a while and in­te­grate them with my cur­rent epistemic prac­tices; thank you for writ­ing this post, I hope you write more like it.

• philoso­phers an­swer clean ques­tions that can be set­tled with for­mal ar­gu­ment

ಠ_ಠ

I think the ex­am­ple of philos­o­phy might ac­tu­ally be a good case for ex­plor­ing prob­lems with this model of dis­agree­ment. Par­tic­u­larly when peo­ple start ar­gu­ing from high-level in­tu­itions that they nonethe­less have difficulty break­ing down, or when at least one of the peo­ple ar­gu­ing is in fact deeply con­fused and is go­ing to say some things that turn out to be non­sense, or when break­ing down the prob­lem in a par­tic­u­lar way im­plic­itly en­dorses cer­tain back­ground as­sump­tions. Pro­duc­tively mak­ing use of dis­agree­ments un­der these con­di­tions seems, to me, to re­quire be­ing will­ing to try out lots of toy mod­els, at­tempt­ing to “quaran­tine” the in­tu­itions un­der dis­pute and tem­porar­ily ig­nore their fruits, and try­ing to find ways to dis­solve the dis­agree­ment or ren­der it moot.

• I switched back to “math­e­mat­i­ci­ans,” I think philoso­phers do this some­times but they also do other stuff so it’s not a good ex­am­ple.

I think philos­o­phy is a fine use case for this ap­proach, it’s messy but ev­ery­thing ends up be­ing messy.

• Any ques­tion that can be an­swered by this pro­ce­dure could even­tu­ally be an­swered us­ing econ 101 di­rectly.

What do you mean by this? Is the idea that you could cre­ate a su­per com­plex (and ac­cu­rate) model, com­posed of only econ 101 parts?

I guess this is true if econ 101 is uni­ver­sal in some sense, so that all 201, etc. in­tu­itions are im­plicit in econ 101 in­tu­itions. Is that what you have in mind?

EDIT: I should have just kept read­ing:

This pro­cess won’t re­veal any truths be­yond those im­plicit in the econ 101 as­sump­tions, but it can do a good job of effi­ciently ex­plor­ing the log­i­cal con­se­quences of those as­sump­tions.
• Alice and Bob’s ar­gu­ment can have loops, if e.g. Alice be­lieve X be­cause of Y, which she be­lieves be­cause of X. We can un­wind these loops by tag­ging an­swers ex­plic­itly with the “depth” of rea­son­ing sup­port­ing that an­swer

A situ­a­tion I’ve come across is that peo­ple of­ten can’t re­mem­ber all the ev­i­dence they used to ar­rive at con­clu­sion X. They re­mem­ber that they spent hours re­search­ing the ques­tion, that they did their best to get bal­anced ev­i­dence and are happy that they con­clu­sion they drew at the time was a fair re­flec­tion of the ev­i­dence they found, but they can’t re­mem­ber the de­tails of the ac­tual re­search, nor are the con­fi­dence that they could re-cre­ate the pro­cess in such a way as to re­dis­cover the ex­act same sub-set of ev­i­dence their search found at that time.

This makes ask­ing them to provide a com­plete list of Ys upon which their X de­pends prob­le­matic, and un­der­stand­ably they feel it is un­fair to ask them to aban­don X, with­out com­pen­sat­ing them for the time to recre­ate an ev­i­den­tial ba­sis equal in size to their ini­tial re­search, or de­mand an equiv­a­lent effort from those op­pos­ing them.

(Note: I’m talk­ing here about what they feel in that situ­a­tion, not what is nec­es­sar­ily ra­tio­nal or fair for them to de­mand.)

• If, in­stead of ask­ing the ques­tion “How do we know what we know?”, we ask in­stead “How re­li­able is knowl­edge that’s de­rived ac­cord­ing to a par­tic­u­lar pro­cess?” then it might be some­thing that could be ob­jec­tively tested, de­spite there be­ing an el­e­ment of self-refer­en­tial­ity (or boot strap­ping) in the as­sump­tion that this sort of test­ing pro­cess is some­thing that can lead to a net in­crease of what we re­li­ably know.

How­ever do­ing so de­pends upon us be­ing able to define the knowl­edge deriva­tion pro­cesses be­ing ex­am­ined pre­cisely enough that ev­i­dence of how they fare in one situ­a­tion is ap­pli­ca­ble to their use in other situ­a­tions, and upon the con­cept of there be­ing a fair way to ob­tain a ran­dom sam­ple of all pos­si­ble situ­a­tions to which they might be ap­plied, de­spite other con­straints upon the ex­am­ple se­lec­tion (such as hav­ing a body of prior knowl­edge against which the test re­sult can be com­pared in or­der to rate the re­li­a­bil­ity of the par­tic­u­lar knowl­edge deriva­tion pro­cess be­ing tested).

De­spite that, if we are look­ing at two ap­proaches to the ques­tion “how much should we in­fer from the differ­ence be­tween chimps and hu­mans?”, we could do worse than spec­ify each ap­proach in a well defined way that is also gen­eral enough to ap­ply to some other situ­a­tions, and then have a third party (that’s ig­no­rant of the spe­cific ap­proaches to be tested) come up with sev­eral test cases with known out­comes, that the two ap­proaches could both be ap­plied to, to see which of them comes up with the more ac­cu­rate pre­dic­tions for a ma­jor­ity of the test cases.