True Sources of Disagreement

Fol­lowup to: Is That Your True Re­jec­tion?

I ex­pected from the be­gin­ning, that the difficult part of two ra­tio­nal­ists rec­on­cil­ing a per­sis­tent dis­agree­ment, would be for them to ex­pose the true sources of their be­liefs.

One sus­pects that this will only work if each party takes re­spon­si­bil­ity for their own end; it’s very hard to see in­side some­one else’s head. Yes­ter­day I ex­hausted my­self men­tally while out on my daily walk, ask­ing my­self the Ques­tion “What do you think you know, and why do you think you know it?” with re­spect to “How much of the AI prob­lem com­presses to large in­sights, and how much of it is un­avoid­able nitty-gritty?” Try­ing to ei­ther un­der­stand why my brain be­lieved what it be­lieved, or else force my brain to ex­pe­rience enough gen­uine doubt that I could re­con­sider the ques­tion and ar­rive at a real jus­tifi­ca­tion that way. It’s hard to see how Robin Han­son could have done any of this work for me.

Pre­sum­ably a sym­met­ri­cal fact holds about my lack of ac­cess to the real rea­sons why Robin be­lieves what he be­lieves. To un­der­stand the true source of a dis­agree­ment, you have to know why both sides be­lieve what they be­lieve—one rea­son why dis­agree­ments are hard to re­solve.

Nonethe­less, here’s my guess as to what this Disagree­ment is about:

If I had to pin­point a sin­gle thing that strikes me as “dis­agree-able” about the way Robin frames his analy­ses, it’s that there are a lot of opaque agents run­ning around, lit­tle black boxes as­sumed to be similar to hu­mans, but there are more of them and they’re less ex­pen­sive to build/​teach/​run. They aren’t even any faster, let alone smarter. (I don’t think that stan­dard eco­nomics says that dou­bling the pop­u­la­tion halves the dou­bling time, so it mat­ters whether you’re mak­ing more minds or faster ones.)

This is Robin’s model for up­loads/​ems, and his model for AIs doesn’t seem to look any differ­ent. So that world looks like this one, ex­cept that the cost of “hu­man cap­i­tal” and la­bor is drop­ping ac­cord­ing to (ex­oge­nous) Moore’s Law , and it ends up that eco­nomic growth dou­bles ev­ery month in­stead of ev­ery six­teen years—but that’s it. Be­ing, my­self, not an economist, this does look to me like a view­point with a dis­tinctly eco­nomic zeit­geist.

In my world, you look in­side the black box. (And, to be sym­met­ri­cal, I don’t spend much time think­ing about more than one box at a time—if I have more hard­ware, it means I have to figure out how to scale a big­ger brain.)

The hu­man brain is a hap­haz­ard thing, thrown to­gether by idiot evolu­tion, as an in­cre­men­tal layer of ic­ing on a chim­panzee cake that never evolved to be gen­er­ally in­tel­li­gent, adapted in a dis­tant world de­void of elab­o­rate sci­en­tific ar­gu­ments or com­puter pro­grams or pro­fes­sional spe­cial­iza­tions.

It’s amaz­ing we can get any­where us­ing the damn thing. But it’s worth re­mem­ber­ing that if there were any smaller mod­ifi­ca­tion of a chim­panzee that spon­ta­neously gave rise to a tech­nolog­i­cal civ­i­liza­tion, we would be hav­ing this con­ver­sa­tion at that lower level of in­tel­li­gence in­stead.

Hu­man neu­rons run at less than a mil­lionth the speed of tran­sis­tors, trans­mit spikes at less than a mil­lionth the speed of light, and dis­si­pate around a mil­lion times the heat per synap­tic op­er­a­tion as the ther­mo­dy­namic min­i­mum for a one-bit op­er­a­tion at room tem­per­a­ture. Phys­i­cally speak­ing, it ought to be pos­si­ble to run a brain at a mil­lion times the speed with­out shrink­ing it, cool­ing it, or in­vok­ing re­versible com­put­ing or quan­tum com­put­ing.

There’s no rea­son to think that the brain’s soft­ware is any closer to the limits of the pos­si­ble than its hard­ware, and in­deed, if you’ve been fol­low­ing along on Over­com­ing Bias this whole time, you should be well aware of the man­i­fold known ways in which our high-level thought pro­cesses fum­ble even the sim­plest prob­lems.

Most of these are not deep, in­her­ent flaws of in­tel­li­gence, or limits of what you can do with a mere hun­dred trillion com­put­ing el­e­ments. They are the re­sults of a re­ally stupid pro­cess that de­signed the retina back­ward, slap­ping to­gether a brain we now use in con­texts way out­side its an­ces­tral en­vi­ron­ment.

Ten thou­sand re­searchers work­ing for one year can­not do the same work as a hun­dred re­searchers work­ing for a hun­dred years; a chim­panzee is one-fourth the vol­ume of a hu­man’s but four chimps do not equal one hu­man; a chim­panzee shares 95% of our DNA but a chim­panzee can­not un­der­stand 95% of what a hu­man can. The scal­ing law for pop­u­la­tion is not the scal­ing law for time is not the scal­ing law for brain size is not the scal­ing law for mind de­sign.

There’s a parable I some­times use, about how the first repli­ca­tor was not quite the end of the era of sta­ble ac­ci­dents, be­cause the pat­tern of the first repli­ca­tor was, of ne­ces­sity, some­thing that could hap­pen by ac­ci­dent. It is only the sec­ond repli­cat­ing pat­tern that you would never have seen with­out many copies of the first repli­ca­tor around to give birth to it; only the sec­ond repli­ca­tor that was part of the world of evolu­tion, some­thing you wouldn’t see in a world of ac­ci­dents.

That first repli­ca­tor must have looked like one of the most bizarre things in the whole his­tory of time—this repli­ca­tor cre­ated purely by chance. But the his­tory of time could never have been set in mo­tion, oth­er­wise.

And what a bizarre thing a hu­man must be, a mind born en­tirely of evolu­tion, a mind that was not cre­ated by an­other mind.

We haven’t yet be­gun to see the shape of the era of in­tel­li­gence.

Most of the uni­verse is far more ex­treme than this gen­tle place, Earth’s cra­dle. Cold vac­uum or the in­te­rior of stars; ei­ther is far more com­mon than the tem­per­ate weather of Earth’s sur­face, where life first arose, in the bal­ance be­tween the ex­tremes. And most pos­si­ble in­tel­li­gences are not bal­anced, like these first hu­mans, in that strange small re­gion of tem­per­ate weather be­tween an amoeba and a Jupiter Brain.

This is the challenge of my own pro­fes­sion—to break your­self loose of the tiny hu­man dot in mind de­sign space, in which we have lived our whole lives, our imag­i­na­tions lul­led to sleep by too-nar­row ex­pe­riences.

For ex­am­ple, Robin says:

Eliezer guesses that within a few weeks a sin­gle AI could grow via largely in­ter­nal means from weak and un­no­ticed to so strong it takes over the world [his ital­ics]

I sup­pose that to a hu­man a “week” sounds like a tem­po­ral con­stant de­scribing a “short pe­riod of time”, but it’s ac­tu­ally 10^49 Planck in­ter­vals, or enough time for a pop­u­la­tion of 2GHz pro­ces­sor cores to perform 10^15 se­rial op­er­a­tions one af­ter the other.

Per­haps the the­sis would sound less shock­ing if Robin had said, “Eliezer guesses that 10^15 se­quen­tial op­er­a­tions might be enough to...”

One should also bear in mind that the hu­man brain, which is not de­signed for the pri­mary pur­pose of sci­en­tific in­sights, does not spend its power effi­ciently on hav­ing many in­sights in min­i­mum time, but this is­sue is harder to un­der­stand than CPU clock speeds.

Robin says he doesn’t like “un­vet­ted ab­strac­tions”. Okay. That’s a strong point. I get it. Un­vet­ted ab­strac­tions go ker­plooie, yes they do in­deed. But some­thing’s wrong with us­ing that as a jus­tifi­ca­tion for mod­els where there are lots of lit­tle black boxes just like hu­mans scur­ry­ing around, and we never pry open the black box and scale the brain big­ger or re­design its soft­ware or even just speed up the damn thing. The in­ter­est­ing part of the prob­lem is harder to an­a­lyze, yes—more dis­tant from the safety rails of over­whelming ev­i­dence—but this is no ex­cuse for re­fus­ing to take it into ac­count.

And in truth I do sus­pect that a strict policy against “un­vet­ted ab­strac­tions” is not the real is­sue here. I con­structed a sim­ple model of an up­load civ­i­liza­tion run­ning on the com­put­ers their econ­omy cre­ates: If a non-up­load civ­i­liza­tion has an ex­po­nen­tial Moore’s Law, y = e^t, then, naively, an up­load civ­i­liza­tion ought to have dy/​dt = e^y → y = -ln(C—t). Not nec­es­sar­ily up to in­finity, but for as long as Moore’s Law would oth­er­wise stay ex­po­nen­tial in a biolog­i­cal civ­i­liza­tion. I walked though the im­pli­ca­tions of this model, show­ing that in many senses it be­haves “just like we would ex­pect” for de­scribing a civ­i­liza­tion run­ning on its own com­put­ers.

Com­pare this to Robin Han­son’s “Eco­nomic Growth Given Ma­chine In­tel­li­gence”, which Robin de­scribes as us­ing “one of the sim­plest en­doge­nous growth mod­els to ex­plore how Moore’s Law changes with com­puter-based work­ers. It is an early but crude at­tempt, but it is the sort of ap­proach I think promis­ing.” Take a quick look at that pa­per.

Now, con­sider the ab­strac­tions used in my Moore’s Re­searchers sce­nario, ver­sus the ab­strac­tions used in Han­son’s pa­per above, and ask your­self only the ques­tion of which looks more “vet­ted by ex­pe­rience”—given that both are mod­els of a sort that haven’t been used be­fore, in do­mains not ac­tu­ally ob­served, and that both give re­sults quite differ­ent from the world we see and that would prob­a­bly cause the vast ma­jor­ity of ac­tual economists to say “Naaaah.”

Moore’s Re­searchers ver­sus Eco­nomic Growth Given Ma­chine In­tel­li­gence—if you didn’t think about the con­clu­sions in ad­vance of the rea­son­ing; and if you also ne­glected that one of these has been writ­ten up in a way that is more im­pres­sive to eco­nomics jour­nals; and you just asked the ques­tion, “To what ex­tent is the math used here, con­strained by our prior ex­pe­rience?” then I would think that the race would at best be even. Or pos­si­bly fa­vor­ing “Moore’s Re­searchers” as be­ing more sim­ple and in­tu­itive, and in­volv­ing less novel math as mea­sured in ad­di­tional quan­tities and laws in­tro­duced.

I ask in all hu­mil­ity if Robin’s true re­jec­tion is a strictly even­hand­edly ap­plied rule that re­jects un­vet­ted ab­strac­tions. Or if, in fact, Robin finds my con­clu­sions, and the sort of premises I use, to be ob­jec­tion­able for other rea­sons—which, so far as we know at this point, may well be valid ob­jec­tions—and so it ap­pears to him that my ab­strac­tions bear a larger bur­den of proof than the sort of math­e­mat­i­cal steps he takes in “Eco­nomic Growth Given Ma­chine In­tel­li­gence”. But rather than offer­ing the rea­sons why the bur­den of proof ap­pears larger to him, he says in­stead that it is “not vet­ted enough”.

One should un­der­stand that “Your ab­strac­tions are un­vet­ted!” makes it difficult for me to en­gage prop­erly. The core of my ar­gu­ment has to do with what hap­pens when you pry open the black boxes that are your eco­nomic agents, and start fid­dling with their brain de­signs, and leave the tiny hu­man dot in mind de­sign space. If all such pos­si­bil­ities are re­jected on the ba­sis of their be­ing “un­vet­ted” by ex­pe­rience, it doesn’t leave me with much to talk about.

Why not just ac­cept the re­jec­tion? Be­cause I ex­pect that to give the wrong an­swer—I ex­pect it to ig­nore the dom­i­nat­ing fac­tor in the Fu­ture, even if the dom­i­nat­ing fac­tor is harder to an­a­lyze.

It shouldn’t be sur­pris­ing if a per­sis­tent dis­agree­ment ends up rest­ing on that point where your at­tempt to take into ac­count the other per­son’s view, runs up against some ques­tion of sim­ple fact where, it seems to you, you know that can’t pos­si­bly be right.

For me, that point is reached when try­ing to vi­su­al­ize a model of in­ter­act­ing black boxes that be­have like hu­mans ex­cept they’re cheaper to make. The world, which shat­tered once with the with the first repli­ca­tor, and shat­tered for the sec­ond time with the emer­gence of hu­man in­tel­li­gence, some­how does not shat­ter a third time. Even in the face of blowups of brain size far greater than the size tran­si­tion from chim­panzee brain to hu­man brain; and changes in de­sign far larger than the de­sign tran­si­tion from chim­panzee brains to hu­man brains; and sim­ple se­rial think­ing speeds that are, maybe even right from the be­gin­ning, thou­sands or mil­lions of times faster.

That’s the point where I, hav­ing spent my ca­reer try­ing to look in­side the black box, try­ing to wrap my tiny brain around the rest of mind de­sign space that isn’t like our small re­gion of tem­per­ate weather, just can’t make my­self be­lieve that the Robin-world is re­ally truly ac­tu­ally the way the fu­ture will be.

There are other things that seem like prob­a­ble nodes of dis­agree­ment:

Robin Han­son’s de­scrip­tion of Friendly AI de­vel­op­ment as “to­tal war” that is harm­ful to even dis­cuss, or his de­scrip­tion of a re­al­ized Friendly AI as “a God to rule us all”. Robin must be vi­su­al­iz­ing an in-prac­tice out­come very differ­ent from what I do, and this seems like a likely source of emo­tional fuel for the dis­agree­ment as well.

Con­versely, Robin Han­son seems to ap­prove of a sce­nario where lots of AIs, of ar­bi­trary mo­tives, con­sti­tute the vast part of the eco­nomic pro­duc­tivity of the So­lar Sys­tem, be­cause he thinks that hu­mans will be pro­tected un­der the legacy le­gal sys­tem that grew con­tin­u­ously out of the mod­ern world, and that the AIs will be un­able to co­or­di­nate to transgress the legacy le­gal sys­tem for fear of los­ing their own le­gal pro­tec­tions. I tend to vi­su­al­ize a some­what differ­ent out­come, to put it mildly; and would sym­met­ri­cally be sus­pected of emo­tional un­will­ing­ness to ac­cept that out­come as in­ex­orable.

Robin doesn’t dis­miss Cyc out of hand and even “hearts” it, which im­plies that we have an ex­tremely differ­ent pic­ture of how in­tel­li­gence works.

Like Robin, I’m also feel­ing burned on this con­ver­sa­tion, and I doubt we’ll finish it; but I should write at least two more posts to try to de­scribe what I’ve learned, and some of the rules that I think I’ve been fol­low­ing.