Superintelligence 6: Intelligence explosion kinetics

This is part of a weekly read­ing group on Nick Bostrom’s book, Su­per­in­tel­li­gence. For more in­for­ma­tion about the group, and an in­dex of posts so far see the an­nounce­ment post. For the sched­ule of fu­ture top­ics, see MIRI’s read­ing guide.

Wel­come. This week we dis­cuss the sixth sec­tion in the read­ing guide: In­tel­li­gence ex­plo­sion ki­net­ics. This cor­re­sponds to Chap­ter 4 in the book, of a similar name. This sec­tion is about how fast a hu­man-level ar­tifi­cial in­tel­li­gence might be­come su­per­in­tel­li­gent.

This post sum­ma­rizes the sec­tion, and offers a few rele­vant notes, and ideas for fur­ther in­ves­ti­ga­tion. Some of my own thoughts and ques­tions for dis­cus­sion are in the com­ments.

There is no need to pro­ceed in or­der through this post, or to look at ev­ery­thing. Feel free to jump straight to the dis­cus­sion. Where ap­pli­ca­ble and I re­mem­ber, page num­bers in­di­cate the rough part of the chap­ter that is most re­lated (not nec­es­sar­ily that the chap­ter is be­ing cited for the spe­cific claim).

Read­ing: Chap­ter 4 (p62-77)


  1. Ques­tion: If and when a hu­man-level gen­eral ma­chine in­tel­li­gence is de­vel­oped, how long will it be from then un­til a ma­chine be­comes rad­i­cally su­per­in­tel­li­gent? (p62)

  2. The fol­low­ing figure from p63 illus­trates some im­por­tant fea­tures in Bostrom’s model of the growth of ma­chine in­tel­li­gence. He en­visages ma­chine in­tel­li­gence pass­ing hu­man-level, then at some point reach­ing the level where most in­puts to fur­ther in­tel­li­gence growth come from the AI it­self (‘crossover’), then pass­ing the level where a sin­gle AI sys­tem is as ca­pa­ble as all of hu­man civ­i­liza­tion, then reach­ing ‘strong su­per­in­tel­li­gence’. The shape of the curve is prob­a­bly in­tended an ex­am­ple rather than a pre­dic­tion.

  3. A tran­si­tion from hu­man-level ma­chine in­tel­li­gence to su­per­in­tel­li­gence might be cat­e­go­rized into one of three sce­nar­ios: ‘slow take­off’ takes decades or cen­turies, ‘mod­er­ate take­off’ takes months or years and ‘fast take­off’ takes min­utes to days. Which sce­nario oc­curs has im­pli­ca­tions for the kinds of re­sponses that might be fea­si­ble.

  4. We can model im­prove­ment in a sys­tem’s in­tel­li­gence with this equa­tion:

    Rate of change in in­tel­li­gence = Op­ti­miza­tion power/​Re­calc­i­trance­

    where ‘op­ti­miza­tion power’ is effort be­ing ap­plied to the prob­lem, and ‘re­calc­i­trance’ is how hard it is to make the sys­tem smarter by ap­ply­ing effort.

  5. Bostrom’s com­ments on re­calc­i­trance of differ­ent meth­ods of in­creas­ing kinds of in­tel­li­gence:

    1. Cog­ni­tive en­hance­ment via pub­lic health and diet: steeply diminish­ing re­turns (i.e. in­creas­ing re­calc­i­trance)

    2. Phar­ma­colog­i­cal en­hancers: diminish­ing re­turns, but per­haps there are still some easy wins be­cause it hasn’t had a lot of at­ten­tion.

    3. Ge­netic cog­ni­tive en­hance­ment: U-shaped re­calc­i­trance—im­prove­ment will be­come eas­ier as meth­ods im­prove, but then re­turns will de­cline. Over­all rates of growth are limited by mat­u­ra­tion tak­ing time.

    4. Net­works and or­ga­ni­za­tions: for or­ga­ni­za­tions as a whole re­calc­i­trance is high. A vast amount of effort is spent on this, and the world only be­comes around a cou­ple of per­cent more pro­duc­tive per year. The in­ter­net may have merely mod­er­ate re­calc­i­trance, but this will likely in­crease as low-hang­ing fruits are de­pleted.

    5. Whole brain em­u­la­tion: re­calc­i­trance is hard to eval­u­ate, but em­u­la­tion of an in­sect will make the path much clearer. After hu­man-level em­u­la­tions ar­rive, re­calc­i­trance will prob­a­bly fall, e.g. be­cause soft­ware ma­nipu­la­tion tech­niques will re­place phys­i­cal-cap­i­tal in­ten­sive scan­ning and image in­ter­pre­ta­tion efforts as the pri­mary ways to im­prove the in­tel­li­gence of the sys­tem. Also there will be new op­por­tu­ni­ties for or­ga­niz­ing the new crea­tures. Even­tu­ally diminish­ing re­turns will set in for these things. Restric­tive reg­u­la­tions might in­crease re­calc­i­trance.

    6. AI al­gorithms: re­calc­i­trance is hard to judge. It could be very low if a sin­gle last key in­sight is dis­cov­ered when much else is ready. Over­all re­calc­i­trance may drop abruptly if a low-re­calc­i­trance sys­tem moves out ahead of higher re­calc­i­trance sys­tems as the most effec­tive method for solv­ing cer­tain prob­lems. We might over­es­ti­mate the re­calc­i­trance of sub-hu­man sys­tems in gen­eral if we see them all as just ‘stupid’.

    7. AI ‘con­tent’: re­calc­i­trance might be very low be­cause of the con­tent already pro­duced by hu­man civ­i­liza­tion, e.g. a smart AI might read the whole in­ter­net fast, and so be­come much bet­ter.

    8. Hard­ware (for AI or up­loads): po­ten­tially low re­calc­i­trance. A pro­ject might be scaled up by or­ders of mag­ni­tude by just pur­chas­ing more hard­ware. In the longer run, hard­ware tends to im­prove ac­cord­ing to Moore’s law, and the in­stalled ca­pac­ity might grow quickly if prices rise due to a de­mand spike from AI.

  6. Op­ti­miza­tion power will prob­a­bly in­crease af­ter AI reaches hu­man-level, be­cause its newfound ca­pa­bil­ities will at­tract in­ter­est and in­vest­ment.

  7. Op­ti­miza­tion power would in­crease more rapidly if AI reaches the ‘crossover’ point, when much of the op­ti­miza­tion power is com­ing from the AI it­self. Be­cause smarter ma­chines can im­prove their in­tel­li­gence more than less smart ma­chines, af­ter the crossover a ‘re­cur­sive self im­prove­ment’ feed­back loop would kick in.

  8. Thus op­ti­miza­tion power is likely to in­crease dur­ing the take­off, and this alone could pro­duce a fast or medium take­off. Fur­ther, re­calc­i­trance is likely to de­cline. Bostrom con­cludes that a fast or medium take­off looks likely, though a slow take­off can­not be ex­cluded.


1. The ar­gu­ment for a rel­a­tively fast take­off is one of the most con­tro­ver­sial ar­gu­ments in the book, so it de­serves some thought. Here is my some­what for­mal­ized sum­mary of the ar­gu­ment as it is pre­sented in this chap­ter. I per­son­ally don’t think it holds, so tell me if that’s be­cause I’m failing to do it jus­tice. The pink bits are not ex­plic­itly in the chap­ter, but are as­sump­tions the ar­gu­ment seems to use.

  1. Growth in in­tel­li­gence = op­ti­miza­tion power /​ re­calc­i­trance [true by defi­ni­tion]

  2. Re­calc­i­trance of AI re­search will prob­a­bly drop or be steady when AI reaches hu­man-level (p68-73)

  3. Op­ti­miza­tion power spent on AI re­search will in­crease af­ter AI reaches hu­man level (p73-77)

  4. Op­ti­miza­tion/​Re­calc­i­trance will stay similarly high for a while prior to crossover

  5. A ‘high’ O/​R ra­tio prior to crossover will pro­duce ex­plo­sive growth OR crossover is close

  6. Within min­utes to years, hu­man-level in­tel­li­gence will reach crossover [from 1-5]

  7. Op­ti­miza­tion power will climb ever faster af­ter crossover, in line with the AI’s own grow­ing ca­pac­ity (p74)

  8. Re­calc­i­trance will not grow much be­tween crossover and superintelligence

  9. Within min­utes to years, crossover-level in­tel­li­gence will reach su­per­in­tel­li­gence [from 7 and 8]

  10. Within min­utes to years, hu­man-level AI will likely tran­si­tion to su­per­in­tel­li­gence [from 6 and 9]

Do you find this com­pel­ling? Should I have filled out the as­sump­tions differ­ently?


2. Other takes on the fast take­off

It seems to me that 5 above is the most con­tro­ver­sial point. The fa­mous Foom De­bate was a long ar­gu­ment be­tween Eliezer Yud­kowsky and Robin Han­son over the plau­si­bil­ity of fast take­off, among other things. Their ar­gu­ments were mostly about both arms of 5, as well as the like­li­hood of an AI tak­ing over the world (to be dis­cussed in a fu­ture week). The Foom De­bate in­cluded a live ver­bal com­po­nent at Jane Street Cap­i­tal: blog sum­mary, video, tran­script. Han­son more re­cently re­viewed Su­per­in­tel­li­gence, again crit­i­ciz­ing the plau­si­bil­ity of a sin­gle pro­ject quickly match­ing the ca­pac­ity of the world.

Kevin Kelly crit­i­cizes point 5 from a differ­ent an­gle: he thinks that speed­ing up hu­man thought can’t speed up progress all that much, be­cause progress will quickly bot­tle­neck on slower pro­cesses.

Others have com­piled lists of crit­i­cisms and de­bates here and here.

3. A closer look at ‘crossover’

Crossover is ‘a point be­yond which the sys­tem’s fur­ther im­prove­ment is mainly driven by the sys­tem’s own ac­tions rather than by work performed upon it by oth­ers’. Another way to put this, avoid­ing cer­tain am­bi­gui­ties, is ‘a point at which the in­puts to a pro­ject are mostly its own out­puts’, such that im­prove­ments to its out­puts feed back into its in­puts.

The na­ture and lo­ca­tion of such a point seems an in­ter­est­ing and im­por­tant ques­tion. If you think crossover is likely to be very nearby for AI, then you need only worry about the re­cur­sive self-im­prove­ment part of the story, which kicks in af­ter crossover. If you think it will be very hard for an AI pro­ject to pro­duce most of its own in­puts, you may want to pay more at­ten­tion to the ar­gu­ments about fast progress be­fore that point.

To have a con­crete pic­ture of crossover, con­sider Google. Sup­pose Google im­proves their search product such that one can find a thing on the in­ter­net a rad­i­cal 10% faster. This makes Google’s own work more effec­tive, be­cause peo­ple at Google look for things on the in­ter­net some­times. How much more effec­tive does this make Google over­all? Maybe they spend a cou­ple of min­utes a day do­ing Google searches, i.e. 0.5% of their work hours, for an over­all sav­ing of .05% of work time. This sug­gests their next im­prove­ments made at Google will be made 1.0005 faster than the last. It will take a while for this pos­i­tive feed­back to take off. If Google co­or­di­nated your eat­ing and or­ga­nized your thoughts and drove your car for you and so on, and then Google im­proved effi­ciency us­ing all of those ser­vices by 10% in one go, then this might make their em­ploy­ees close to 10% more pro­duc­tive, which might pro­duce more no­tice­able feed­back. Then Google would have reached the crossover. This is per­haps eas­ier to imag­ine for Google than other pro­jects, yet I think still fairly hard to imag­ine.

Han­son talks more about this is­sue when he asks why the ex­plo­sion ar­gu­ment doesn’t ap­ply to other re­cur­sive tools. He points to Dou­glas En­gle­bart’s am­bi­tious pro­posal to use com­puter tech­nolo­gies to pro­duce a rapidly self-im­prov­ing tool set.

Below is a sim­ple model of a pro­ject which con­tributes all of its own in­puts, and one which be­gins mostly be­ing im­proved by the world. They are both nor­mal­ized to be­gin one tenth as large as the world and to grow at the same pace as each other (this is why the one with help grows slower, per­haps coun­ter­in­tu­itively). As you can see, the pro­ject which is re­spon­si­ble for its own im­prove­ment takes far less time to reach its ‘sin­gu­lar­ity’, and is more abrupt. It starts out at crossover. The pro­ject which is helped by the world doesn’t reach crossover un­til it passes 1.

4. How much differ­ence does at­ten­tion and fund­ing make to re­search?

In­ter­est and in­vest­ments in AI at around hu­man-level are (nat­u­rally) hy­poth­e­sized to ac­cel­er­ate AI de­vel­op­ment in this chap­ter. It would be good to have more em­piri­cal ev­i­dence on the quan­ti­ta­tive size of such an effect. I’ll start with one ex­am­ple, be­cause ex­am­ples are a bit costly to in­ves­ti­gate. I se­lected re­new­able en­ergy be­fore I knew the re­sults, be­cause they come up early in the Perfor­mance Curves Database, and I thought their fund­ing likely to have been un­sta­ble. In­deed, OECD fund­ing since the 70s looks like this ap­par­ently:

(from here)

The steep in­crease in fund­ing in the early 80s was due to Pres­i­dent Carter’s en­ergy poli­cies, which were re­lated to the 1979 oil crisis.

This is what var­i­ous in­di­ca­tors of progress in re­new­able en­er­gies look like (click on them to see their sources):

There are quite a few more at the Perfor­mance Curves Database. I see sur­pris­ingly lit­tle re­la­tion­ship be­tween the fund­ing curves and these met­rics of progress. Some of them are shock­ingly straight. What is go­ing on? (I haven’t looked into these more than you see here).

5. Other writ­ings on re­cur­sive self-improvement

Eliezer Yud­kowsky wrote about the idea origi­nally, e.g. here. David Chalmers in­ves­ti­gated the topic in some de­tail, and Mar­cus Hut­ter did some more. More poin­t­ers here.

In-depth investigations

If you are par­tic­u­larly in­ter­ested in these top­ics, and want to do fur­ther re­search, these are a few plau­si­ble di­rec­tions, some in­spired by Luke Muehlhauser’s list, which con­tains many sug­ges­tions re­lated to parts of Su­per­in­tel­li­gence. Th­ese pro­jects could be at­tempted at var­i­ous lev­els of depth.

  1. Model the in­tel­li­gence ex­plo­sion more pre­cisely. Take in­spira­tion from suc­cess­ful eco­nomic mod­els, and ev­i­dence from a wide range of em­piri­cal ar­eas such as evolu­tion­ary biol­ogy, tech­nolog­i­cal his­tory, al­gorith­mic progress, and ob­served tech­nolog­i­cal trends. Eliezer Yud­kowsky has writ­ten at length about this pro­ject.

  2. Es­ti­mate em­piri­cally a spe­cific in­ter­ac­tion in the in­tel­li­gence ex­plo­sion model. For in­stance, how much and how quickly does in­vest­ment in­crease in tech­nolo­gies that look promis­ing? How much differ­ence does that make to the rate of progress in the tech­nol­ogy? How much does scal­ing up re­searchers change out­put in com­puter sci­ence? (Rele­vant to how much adding ex­tra ar­tifi­cial AI re­searchers speeds up progress) How much do con­tem­po­rary or­ga­ni­za­tions con­tribute to their own in­puts? (i.e. how hard would it be for a pro­ject to con­tribute more to its own in­puts than the rest of the world put to­gether, such that a sub­stan­tial pos­i­tive feed­back might en­sue?) Yud­kowsky 2013 again has a few poin­t­ers (e.g. start­ing at p15).

  3. If hu­man thought was sped up sub­stan­tially, what would be the main limits to ar­bi­trar­ily fast tech­nolog­i­cal progress?

    If you are in­ter­ested in any­thing like this, you might want to men­tion it in the com­ments, and see whether other peo­ple have use­ful thoughts.

    How to proceed

    This has been a col­lec­tion of notes on the chap­ter. The most im­por­tant part of the read­ing group though is dis­cus­sion, which is in the com­ments sec­tion. I pose some ques­tions for you there, and I in­vite you to add your own. Please re­mem­ber that this group con­tains a va­ri­ety of lev­els of ex­per­tise: if a line of dis­cus­sion seems too ba­sic or too in­com­pre­hen­si­ble, look around for one that suits you bet­ter!

    Next week, we will talk about ‘de­ci­sive strate­gic ad­van­tage’: the pos­si­bil­ity of a sin­gle AI pro­ject get­ting huge amounts of power in an AI tran­si­tion. To pre­pare, read Chap­ter 5, De­ci­sive Strate­gic Ad­van­tage (p78-90). The dis­cus­sion will go live at 6pm Pa­cific time next Mon­day Oct 27. Sign up to be no­tified here.