Hard Takeoff

Con­tinu­a­tion of: Re­cur­sive Self-Improvement

Con­stant nat­u­ral se­lec­tion pres­sure, op­er­at­ing on the genes of the ho­minid line, pro­duced im­prove­ment in brains over time that seems to have been, roughly, lin­ear or ac­cel­er­at­ing; the op­er­a­tion of con­stant hu­man brains on a pool of knowl­edge seems to have pro­duced re­turns that are, very roughly, ex­po­nen­tial or su­per­ex­po­nen­tial. (Robin pro­poses that hu­man progress is well-char­ac­ter­ized as a se­ries of ex­po­nen­tial modes with diminish­ing dou­bling times.)

Re­cur­sive self-im­prove­ment—an AI rewrit­ing its own cog­ni­tive al­gorithms—iden­ti­fies the ob­ject level of the AI with a force act­ing on the metacog­ni­tive level; it “closes the loop” or “folds the graph in on it­self”. E.g. the differ­ence be­tween re­turns on a con­stant in­vest­ment in a bond, and rein­vest­ing the re­turns into pur­chas­ing fur­ther bonds, is the differ­ence be­tween the equa­tions y = f(t) = m*t, and dy/​dt = f(y) = m*y whose solu­tion is the com­pound in­ter­est ex­po­nen­tial, y = e^(m*t).

When you fold a whole chain of differ­en­tial equa­tions in on it­self like this, it should ei­ther pe­ter out rapidly as im­prove­ments fail to yield fur­ther im­prove­ments, or else go FOOM. An ex­actly right law of diminish­ing re­turns that lets the sys­tem fly through the soft take­off key­hole is un­likely—far more un­likely than see­ing such be­hav­ior in a sys­tem with a roughly-con­stant un­der­ly­ing op­ti­mizer, like evolu­tion im­prov­ing brains, or hu­man brains im­prov­ing tech­nol­ogy. Our pre­sent life is no good in­di­ca­tor of things to come.

Or to try and com­press it down to a slo­gan that fits on a T-Shirt—not that I’m say­ing this is a good idea—“Moore’s Law is ex­po­nen­tial now; it would be re­ally odd if it stayed ex­po­nen­tial with the im­prov­ing com­put­ers do­ing the re­search.” I’m not say­ing you liter­ally get dy/​dt = e^y that goes to in­finity af­ter finite time—and hard­ware im­prove­ment is in some ways the least in­ter­est­ing fac­tor here—but should we re­ally see the same curve we do now?

RSI is the biggest, most in­ter­est­ing, hard­est-to-an­a­lyze, sharpest break-with-the-past con­tribut­ing to the no­tion of a “hard take­off” aka “AI go FOOM”, but it’s nowhere near be­ing the only such fac­tor. The ad­vent of hu­man in­tel­li­gence was a dis­con­ti­nu­ity with the past even with­out RSI...

...which is to say that ob­served evolu­tion­ary his­tory—the dis­con­ti­nu­ity be­tween hu­mans, and chimps who share 95% of our DNA—lightly sug­gests a crit­i­cal thresh­old built into the ca­pa­bil­ities that we think of as “gen­eral in­tel­li­gence”, a ma­chine that be­comes far more pow­er­ful once the last gear is added.

This is only a light sug­ges­tion be­cause the branch­ing time be­tween hu­mans and chimps is enough time for a good deal of com­plex adap­ta­tion to oc­cur. We could be look­ing at the sum of a cas­cade, not the ad­di­tion of a fi­nal miss­ing gear. On the other hand, we can look at the gross brain anatomies and see that hu­man brain anatomy and chimp anatomy have not di­verged all that much. On the grip­ping hand, there’s the sud­den cul­tural rev­olu­tion—the sud­den in­crease in the so­phis­ti­ca­tion of ar­ti­facts—that ac­com­panied the ap­pear­ance of anatom­i­cally Cro-Magnons just a few tens of thou­sands of years ago.

Now of course this might all just be com­pletely in­ap­pli­ca­ble to the de­vel­op­ment tra­jec­tory of AIs built by hu­man pro­gram­mers rather than by evolu­tion. But it at least lightly sug­gests, and pro­vides a hy­po­thet­i­cal illus­tra­tion of, a dis­con­tin­u­ous leap up­ward in ca­pa­bil­ity that re­sults from a nat­u­ral fea­ture of the solu­tion space—a point where you go from sorta-okay solu­tions to to­tally-amaz­ing solu­tions as the re­sult of a few fi­nal tweaks to the mind de­sign.

I could po­ten­tially go on about this no­tion for a bit—be­cause, in an evolu­tion­ary tra­jec­tory, it can’t liter­ally be a “miss­ing gear”, the sort of dis­con­ti­nu­ity that fol­lows from re­mov­ing a gear that an oth­er­wise func­tion­ing ma­chine was built around. So if you sup­pose that a fi­nal set of changes was enough to pro­duce a sud­den huge leap in effec­tive in­tel­li­gence, it does de­mand the ques­tion of what those changes were. Some­thing to do with re­flec­tion—the brain mod­el­ing or con­trol­ling it­self—would be one ob­vi­ous can­di­date. Or per­haps a change in mo­ti­va­tions (more cu­ri­ous in­di­vi­d­u­als, us­ing the brain­power they have in differ­ent di­rec­tions) in which case you wouldn’t ex­pect that dis­con­ti­nu­ity to ap­pear in the AI’s de­vel­op­ment, but you would ex­pect it to be more effec­tive at ear­lier stages than hu­man­ity’s evolu­tion­ary his­tory would sug­gest… But you could have whole jour­nal is­sues about that one ques­tion, so I’m just go­ing to leave it at that.

Or con­sider the no­tion of sud­den re­source bo­nan­zas. Sup­pose there’s a semi-so­phis­ti­cated Ar­tifi­cial Gen­eral In­tel­li­gence run­ning on a cluster of a thou­sand CPUs. The AI has not hit a wall—it’s still im­prov­ing it­self—but its self-im­prove­ment is go­ing so slowly that, the AI calcu­lates, it will take an­other fifty years for it to en­g­ineer /​ im­ple­ment /​ re­fine just the changes it cur­rently has in mind. Even if this AI would go FOOM even­tu­ally, its cur­rent progress is so slow as to con­sti­tute be­ing flatlined...

So the AI turns its at­ten­tion to ex­am­in­ing cer­tain blobs of bi­nary code—code com­pos­ing op­er­at­ing sys­tems, or routers, or DNS ser­vices—and then takes over all the poorly defended com­put­ers on the In­ter­net. This may not re­quire what hu­mans would re­gard as ge­nius, just the abil­ity to ex­am­ine lots of ma­chine code and do rel­a­tively low-grade rea­son­ing on mil­lions of bytes of it. (I have a say­ing/​hy­poth­e­sis that a hu­man try­ing to write code is like some­one with­out a vi­sual cor­tex try­ing to paint a pic­ture—we can do it even­tu­ally, but we have to go pixel by pixel be­cause we lack a sen­sory modal­ity for that medium; it’s not our na­tive en­vi­ron­ment.) The Fu­ture may also have more le­gal ways to ob­tain large amounts of com­put­ing power quickly.

This sort of re­source bo­nanza is in­trigu­ing in a num­ber of ways. By as­sump­tion, op­ti­miza­tion effi­ciency is the same, at least for the mo­ment—we’re just plug­ging a few or­ders of mag­ni­tude more re­source into the cur­rent in­put/​out­put curve. With a stupid al­gorithm, a few or­ders of mag­ni­tude more com­put­ing power will buy you only a lin­ear in­crease in perfor­mance—I would not fear Cyc even if ran on a com­puter the size of the Moon, be­cause there is no there there.

On the other hand, hu­mans have a brain three times as large, and a pre­frontal cor­tex six times as large, as that of a stan­dard pri­mate our size—so with soft­ware im­prove­ments of the sort that nat­u­ral se­lec­tion made over the last five mil­lion years, it does not re­quire ex­po­nen­tial in­creases in com­put­ing power to sup­port lin­early greater in­tel­li­gence. Mind you, this sort of biolog­i­cal anal­ogy is always fraught—maybe a hu­man has not much more cog­ni­tive horse­power than a chim­panzee, the same un­der­ly­ing tasks be­ing performed, but in a few more do­mains and with greater re­flec­tivity—the en­g­ine out­puts the same horse­power, but a few gears were re­con­figured to turn each other less waste­fully—and so you wouldn’t be able to go from hu­man to su­per-hu­man with just an­other sixfold in­crease in pro­cess­ing power… or some­thing like that.

But if the les­son of biol­ogy sug­gests any­thing, it is that you do not run into log­a­r­ith­mic re­turns on pro­cess­ing power in the course of reach­ing hu­man in­tel­li­gence, even when that pro­cess­ing power in­crease is strictly par­allel rather than se­rial, pro­vided that you are at least as good as writ­ing soft­ware to take ad­van­tage of that in­creased com­put­ing power, as nat­u­ral se­lec­tion is at pro­duc­ing adap­ta­tions—five mil­lion years for a sixfold in­crease in com­put­ing power.

Michael Vas­sar ob­served in yes­ter­day’s com­ments that hu­mans, by spend­ing lin­early more time study­ing chess, seem to get lin­ear in­creases in their chess rank (across a wide range of rank­ings), while putting ex­po­nen­tially more time into a search al­gorithm is usu­ally re­quired to yield the same range of in­crease. Vas­sar called this “bizarre”, but I find it quite nat­u­ral. Deep Blue searched the raw game tree of chess; Kas­par­avo searched the com­pressed reg­u­lar­i­ties of chess. It’s not sur­pris­ing that the sim­ple al­gorithm is log­a­r­ith­mic and the so­phis­ti­cated al­gorithm is lin­ear. One might say similarly of the course of hu­man progress seem­ing to be closer to ex­po­nen­tial, while evolu­tion­ary progress is closer to be­ing lin­ear. Be­ing able to un­der­stand the reg­u­lar­ity of the search space counts for quite a lot.

If the AI is some­where in be­tween—not as brute-force as Deep Blue, nor as com­pressed as a hu­man—then maybe a 10,000-fold in­crease in com­put­ing power will only buy it a 10-fold in­crease in op­ti­miza­tion ve­loc­ity… but that’s still quite a speedup.

Fur­ther­more, all fu­ture im­prove­ments the AI makes to it­self will now be amor­tized over 10,000 times as much com­put­ing power to ap­ply the al­gorithms. So a sin­gle im­prove­ment to code now has more im­pact than be­fore; it’s li­able to pro­duce more fur­ther im­prove­ments. Think of a ura­nium pile. It’s always run­ning the same “al­gorithm” with re­spect to neu­trons caus­ing fis­sions that pro­duce fur­ther neu­trons, but just piling on more ura­nium can cause it to go from sub­crit­i­cal to su­per­crit­i­cal, as any given neu­tron has more ura­nium to travel through and a higher chance of caus­ing fu­ture fis­sions.

So just the re­source bo­nanza rep­re­sented by “eat­ing the In­ter­net” or “dis­cov­er­ing an ap­pli­ca­tion for which there is effec­tively un­limited de­mand, which lets you rent huge amounts of com­put­ing power while us­ing only half of it to pay the bills”—even though this event isn’t par­tic­u­larly re­cur­sive of it­self, just an ob­ject-level fruit-tak­ing—could po­ten­tially drive the AI from sub­crit­i­cal to su­per­crit­i­cal.

Not, mind you, that this will hap­pen with an AI that’s just stupid. But an AI already im­prov­ing it­self slowly—that’s a differ­ent case.

Even if this doesn’t hap­pen—if the AI uses this newfound com­put­ing power at all effec­tively, its op­ti­miza­tion effi­ciency will in­crease more quickly than be­fore; just be­cause the AI has more op­ti­miza­tion power to ap­ply to the task of in­creas­ing its own effi­ciency, thanks to the sud­den bo­nanza of op­ti­miza­tion re­sources.

So the whole tra­jec­tory can con­ceiv­ably change, just from so sim­ple and straight­for­ward and un­clever and un­in­ter­est­ing-seem­ing an act, as eat­ing the In­ter­net. (Or rent­ing a big­ger cloud.)

Agri­cul­ture changed the course of hu­man his­tory by sup­port­ing a larger pop­u­la­tion—and that was just a ques­tion of hav­ing more hu­mans around, not in­di­vi­d­ual hu­mans hav­ing a brain a hun­dred times as large. This gets us into the whole is­sue of the re­turns on scal­ing in­di­vi­d­ual brains not be­ing any­thing like the re­turns on scal­ing the num­ber of brains. A big-brained hu­man has around four times the cra­nial vol­ume of a chim­panzee, but 4 chimps != 1 hu­man. (And for that mat­ter, 60 squir­rels != 1 chimp.) Soft­ware im­prove­ments here al­most cer­tainly com­pletely dom­i­nate hard­ware, of course. But hav­ing a thou­sand sci­en­tists who col­lec­tively read all the pa­pers in a field, and who talk to each other, is not like hav­ing one su­per­scien­tist who has read all those pa­pers and can cor­re­late their con­tents di­rectly us­ing na­tive cog­ni­tive pro­cesses of as­so­ci­a­tion, recog­ni­tion, and ab­strac­tion. Hav­ing more hu­mans talk­ing to each other us­ing low-band­width words, can­not be ex­pected to achieve re­turns similar to those from scal­ing com­po­nent cog­ni­tive pro­cesses within a co­her­ent cog­ni­tive sys­tem.

This, too, is an idiom out­side hu­man ex­pe­rience—we have to solve big prob­lems us­ing lots of hu­mans, be­cause there is no way to solve them us­ing ONE BIG hu­man. But it never oc­curs to any­one to sub­sti­tute four chimps for one hu­man; and only a cer­tain very fool­ish kind of boss thinks you can sub­sti­tute ten pro­gram­mers with one year of ex­pe­rience for one pro­gram­mer with ten years of ex­pe­rience.

(Part of the gen­eral Cul­ture of Chaos that praises emer­gence and thinks evolu­tion is smarter than hu­man de­sign­ers, also has a mythol­ogy of groups be­ing in­her­ently su­pe­rior to in­di­vi­d­u­als. But this is gen­er­ally a mat­ter of poor in­di­vi­d­ual ra­tio­nal­ity, and var­i­ous ar­cane group struc­tures that are sup­posed to com­pen­sate; rather than an in­her­ent fact about cog­ni­tive pro­cesses some­how scal­ing bet­ter when chopped up into dis­tinct brains. If that were liter­ally more effi­cient, evolu­tion would have de­signed hu­mans to have four chim­panzee heads that ar­gued with each other. In the realm of AI, it seems much more straight­for­ward to have a sin­gle cog­ni­tive pro­cess that lacks the emo­tional stub­born­ness to cling to its ac­cus­tomed the­o­ries, and doesn’t need to be ar­gued out of it at gun­point or re­placed by a new gen­er­a­tion of grad stu­dents. I’m not go­ing to delve into this in de­tail for now, just warn you to be sus­pi­cious of this par­tic­u­lar creed of the Cul­ture of Chaos; it’s not like they ac­tu­ally ob­served the rel­a­tive perfor­mance of a hun­dred hu­mans ver­sus one BIG mind with a brain fifty times hu­man size.)

So yes, there was a lot of soft­ware im­prove­ment in­volved—what we are see­ing with the mod­ern hu­man brain size, is prob­a­bly not so much the brain vol­ume re­quired to sup­port the soft­ware im­prove­ment, but rather the new evolu­tion­ary equil­ibrium for brain size given the im­proved soft­ware.

Even so—ho­minid brain size in­creased by a fac­tor of five over the course of around five mil­lion years. You might want to think very se­ri­ously about the con­trast be­tween that idiom, and a suc­cess­ful AI be­ing able to ex­pand onto five thou­sand times as much hard­ware over the course of five min­utes—when you are pon­der­ing pos­si­ble hard take­offs, and whether the AI tra­jec­tory ought to look similar to hu­man ex­pe­rience.

A sub­tler sort of hard­ware over­hang, I sus­pect, is rep­re­sented by mod­ern CPUs have a 2GHz se­rial speed, in con­trast to neu­rons that spike 100 times per sec­ond on a good day. The “hun­dred-step rule” in com­pu­ta­tional neu­ro­science is a rule of thumb that any pos­tu­lated neu­ral al­gorithm which runs in re­al­time has to perform its job in less than 100 se­rial steps one af­ter the other. We do not un­der­stand how to effi­ciently use the com­puter hard­ware we have now, to do in­tel­li­gent think­ing. But the much-vaunted “mas­sive par­allelism” of the hu­man brain, is, I sus­pect, mostly cache lookups to make up for the sheer awk­ward­ness of the brain’s se­rial slow­ness—if your com­puter ran at 200Hz, you’d have to re­sort to all sorts of ab­surdly mas­sive par­allelism to get any­thing done in re­al­time. I sus­pect that, if cor­rectly de­signed, a mid­size com­puter cluster would be able to get high-grade think­ing done at a se­rial speed much faster than hu­man, even if the to­tal par­allel com­put­ing power was less.

So that’s an­other kind of over­hang: be­cause our com­put­ing hard­ware has run so far ahead of AI the­ory, we have in­cred­ibly fast com­put­ers we don’t know how to use for think­ing; get­ting AI right could pro­duce a huge, dis­con­tin­u­ous jolt, as the speed of high-grade thought on this planet sud­denly dropped into com­puter time.

A still sub­tler kind of over­hang would be rep­re­sented by hu­man failure to use our gath­ered ex­per­i­men­tal data effi­ciently.

On to the topic of in­sight, an­other po­ten­tial source of dis­con­ti­nu­ity. The course of ho­minid evolu­tion was driven by evolu­tion’s neigh­bor­hood search; if the evolu­tion of the brain ac­cel­er­ated to some de­gree, this was prob­a­bly due to ex­ist­ing adap­ta­tions cre­at­ing a greater num­ber of pos­si­bil­ities for fur­ther adap­ta­tions. (But it couldn’t ac­cel­er­ate past a cer­tain point, be­cause evolu­tion is limited in how much se­lec­tion pres­sure it can ap­ply—if some­one suc­ceeds in breed­ing due to adap­ta­tion A, that’s less var­i­ance left over for whether or not they suc­ceed in breed­ing due to adap­ta­tion B.)

But all this is search­ing the raw space of genes. Hu­man de­sign in­tel­li­gence, or suffi­ciently so­phis­ti­cated AI de­sign in­tel­li­gence, isn’t like that. One might even be tempted to make up a com­pletely differ­ent curve out of thin air—like, in­tel­li­gence will take all the easy wins first, and then be left with only higher-hang­ing fruit, while in­creas­ing com­plex­ity will defeat the abil­ity of the de­signer to make changes. So where blind evolu­tion ac­cel­er­ated, in­tel­li­gent de­sign will run into diminish­ing re­turns and grind to a halt. And as long as you’re mak­ing up fairy tales, you might as well fur­ther add that the law of diminish­ing re­turns will be ex­actly right, and have bumps and rough patches in ex­actly the right places, to pro­duce a smooth gen­tle take­off even af­ter re­cur­sion and var­i­ous hard­ware tran­si­tions are fac­tored in… One also won­ders why the story about “in­tel­li­gence tak­ing easy wins first in de­sign­ing brains” tops out at or be­fore hu­man-level brains, rather than go­ing a long way be­yond hu­man be­fore top­ping out. But one sus­pects that if you tell that story, there’s no point in in­vent­ing a law of diminish­ing re­turns to be­gin with.

(Ul­ti­mately, if the char­ac­ter of phys­i­cal law is any­thing like our cur­rent laws of physics, there will be limits to what you can do on finite hard­ware, and limits to how much hard­ware you can as­sem­ble in finite time, but if they are very high limits rel­a­tive to hu­man brains, it doesn’t af­fect the ba­sic pre­dic­tion of hard take­off, “AI go FOOM”.)

The main thing I’ll ven­ture into ac­tu­ally ex­pect­ing from adding “in­sight” to the mix, is that there’ll be a dis­con­ti­nu­ity at the point where the AI un­der­stands how to do AI the­ory, the same way that hu­man re­searchers try to do AI the­ory. An AI, to swal­low its own op­ti­miza­tion chain, must not just be able to rewrite its own source code; it must be able to, say, rewrite Ar­tifi­cial In­tel­li­gence: A Modern Ap­proach (2nd Edi­tion). An abil­ity like this seems (un­trust­worthily, but I don’t know what else to trust) like it ought to ap­pear at around the same time that the ar­chi­tec­ture is at the level of, or ap­proach­ing the level of, be­ing able to han­dle what hu­mans han­dle—be­ing no shal­lower than an ac­tual hu­man, what­ever its in­ex­pe­rience in var­i­ous do­mains. It would pro­duce fur­ther dis­con­ti­nu­ity at around that time.

In other words, when the AI be­comes smart enough to do AI the­ory, that’s when I ex­pect it to fully swal­low its own op­ti­miza­tion chain and for the real FOOM to oc­cur—though the AI might reach this point as part of a cas­cade that started at a more prim­i­tive level.

All these com­pli­ca­tions is why I don’t be­lieve we can re­ally do any sort of math that will pre­dict quan­ti­ta­tively the tra­jec­tory of a hard take­off. You can make up mod­els, but real life is go­ing to in­clude all sorts of dis­crete jumps, bot­tle­necks, bo­nan­zas, in­sights—and the “fold the curve in on it­self” paradigm of re­cur­sion is go­ing to am­plify even small rough­nesses in the tra­jec­tory.

So I stick to qual­i­ta­tive pre­dic­tions. “AI go FOOM”.

To­mor­row I hope to tackle lo­cal­ity, and a bes­tiary of some pos­si­ble qual­i­ta­tive tra­jec­to­ries the AI might take given this anal­y­sis. Robin Han­son’s sum­mary of “prim­i­tive AI fooms to so­phis­ti­cated AI” doesn’t fully rep­re­sent my views—that’s just one en­try in the bes­tiary, albeit a ma­jor one.