Superintelligence 8: Cognitive superpowers

This is part of a weekly read­ing group on Nick Bostrom’s book, Su­per­in­tel­li­gence. For more in­for­ma­tion about the group, and an in­dex of posts so far see the an­nounce­ment post. For the sched­ule of fu­ture top­ics, see MIRI’s read­ing guide.

Wel­come. This week we dis­cuss the eighth sec­tion in the read­ing guide: Cog­ni­tive Su­per­pow­ers. This cor­re­sponds to Chap­ter 6.

This post sum­ma­rizes the sec­tion, and offers a few rele­vant notes, and ideas for fur­ther in­ves­ti­ga­tion. Some of my own thoughts and ques­tions for dis­cus­sion are in the com­ments.

There is no need to pro­ceed in or­der through this post, or to look at ev­ery­thing. Feel free to jump straight to the dis­cus­sion. Where ap­pli­ca­ble and I re­mem­ber, page num­bers in­di­cate the rough part of the chap­ter that is most re­lated (not nec­es­sar­ily that the chap­ter is be­ing cited for the spe­cific claim).

Read­ing: Chap­ter 6


  1. AI agents might have very differ­ent skill pro­files.

  2. AI with some nar­row skills could pro­duce a va­ri­ety of other skills. e.g. strong AI re­search skills might al­low an AI to build its own so­cial skills.

  3. ‘Su­per­pow­ers’ that might be par­tic­u­larly im­por­tant for an AI that wants to take con­trol of the world in­clude:

    1. In­tel­li­gence am­plifi­ca­tion: for boot­strap­ping its own intelligence

    2. Strate­giz­ing: for achiev­ing dis­tant goals and over­com­ing opposition

    3. So­cial ma­nipu­la­tion: for es­cap­ing hu­man con­trol, get­ting sup­port, and en­courag­ing de­sired courses of action

    4. Hack­ing: for steal­ing hard­ware, money and in­fras­truc­ture; for es­cap­ing hu­man control

    5. Tech­nol­ogy re­search: for cre­at­ing mil­i­tary force, surveillance, or space transport

    6. Eco­nomic pro­duc­tivity: for mak­ing money to spend on tak­ing over the world

  4. Th­ese ‘su­per­pow­ers’ are rel­a­tive to other nearby agents; Bostrom means them to be su­per only if they sub­stan­tially ex­ceed the com­bined ca­pa­bil­ities of the rest of the global civ­i­liza­tion.

  5. A takeover sce­nario might go like this:

    1. Pre-crit­i­cal­ity: re­searchers make a seed-AI, which be­comes in­creas­ingly helpful at im­prov­ing itself

    2. Re­cur­sive self-im­prove­ment: seed-AI be­comes main force for im­prov­ing it­self and brings about an in­tel­li­gence ex­plo­sion. It per­haps de­vel­ops all of the su­per­pow­ers it didn’t already have.

    3. Covert prepa­ra­tion: the AI makes up a ro­bust long term plan, pre­tends to be nice, and es­capes from hu­man con­trol if need be.

    4. Overt im­ple­men­ta­tion: the AI goes ahead with its plan, per­haps kil­ling the hu­mans at the out­set to re­move op­po­si­tion.

  6. Wise Sin­gle­ton Sus­tain­abil­ity Thresh­old (WSST): a ca­pa­bil­ity set ex­ceeds this iff a wise sin­gle­ton with that ca­pa­bil­ity set would be able to take over much of the ac­cessible uni­verse. ‘Wise’ here means be­ing pa­tient and savvy about ex­is­ten­tial risks, ‘sin­gle­ton’ means be­ing in­ter­nally co­or­di­nated and hav­ing no op­po­nents.

  7. The WSST ap­pears to be low. e.g. our own in­tel­li­gence is suffi­cient, as would some skill sets be that were strong in only a few nar­row ar­eas.

  8. The cos­mic en­dow­ment (what we could do with the mat­ter and en­ergy that might ul­ti­mately be available if we colonized space) is at least about 10^85 com­pu­ta­tional op­er­a­tions. This is equiv­a­lent to 10^58 em­u­lated hu­man lives.

Another view

Bostrom starts the chap­ter claiming that hu­mans’ dom­i­nant po­si­tion comes from their slightly ex­panded set of cog­ni­tive func­tions rel­a­tive to other an­i­mals. Com­puter sci­en­tist Ernest Davis crit­i­cizes this claim in a re­cent re­view of Su­per­in­tel­li­gence:

The as­sump­tion that a large gain in in­tel­li­gence would nec­es­sar­ily en­tail a cor­re­spond­ingly large in­crease in power. Bostrom points out that what he calls a com­par­a­tively small in­crease in brain size and com­plex­ity re­sulted in mankind’s spec­tac­u­lar gain in phys­i­cal power. But he ig­nores the fact that the much larger in­crease in brain size and com­plex­ity that pre­ceded the ap­pear­ance in man had no such effect. He says that the re­la­tion of a su­per­com­puter to man will be like the re­la­tion of a man to a mouse, rather than like the re­la­tion of Ein­stein to the rest of us; but what if it is like the re­la­tion of an elephant to a mouse?


1. How does this model of AIs with unique bun­dles of ‘su­per­pow­ers’ fit with the story we have heard so far?
Ear­lier it seemed we were just talk­ing about a sin­gle level of in­tel­li­gence that was grow­ing, whereas now it seems there are po­ten­tially many dis­tinct in­tel­li­gent skills that might need to be de­vel­oped. Does our ar­gu­ment so far still work out, if an agent has a va­ri­ety of differ­ent sorts of in­tel­li­gence to be im­prov­ing?
If you re­call, the main ar­gu­ment so far was that AI might be easy (have ‘low re­calc­i­trance’) mostly be­cause there is a lot of hard­ware and con­tent sit­ting around and al­gorithms might ran­domly hap­pen to be easy. Then more effort (‘op­ti­miza­tion power’) will be spent on AI as it be­came ev­i­dently im­por­tant. Then much more effort again will be spent when the AI be­comes a large source of la­bor it­self. This was all taken to sug­gest that AI might progress very fast from hu­man-level to su­per­hu­man level, which sug­gests that one AI agent might get far ahead be­fore any­one else catches up, sug­gest­ing that one AI might seize power.
It seems to me that this ar­gu­ment works a bit less well with a cluster of skills than one cen­tral im­por­tant skill, though it is a mat­ter of de­gree and the ar­gu­ment was only qual­i­ta­tive to be­gin with.
It is less likely that AI al­gorithms will hap­pen to be es­pe­cially easy if a lot of differ­ent al­gorithms are needed. Also, if differ­ent cog­ni­tive skills are de­vel­oped at some­what differ­ent times, then it’s harder to imag­ine a sud­den jump when a fully ca­pa­ble AI sud­denly reads the whole in­ter­net or be­comes a hugely more valuable use for hard­ware than any­thing be­ing run already. Then if there are many differ­ent pro­jects needed for mak­ing an AI smarter in differ­ent ways, the ex­tra effort (brought first by hu­man op­ti­mism and then by self-im­prov­ing AI) must be di­vided be­tween those pro­jects. If a gi­ant AI could ded­i­cate its efforts to im­prov­ing some cen­tral fea­ture that would im­prove all of its fu­ture efforts (like ‘in­tel­li­gence’), then it would do much bet­ter than if it has to de­vote one one thou­sandth of its efforts to each of a thou­sand differ­ent sub-skills, each of which is only rele­vant for a few niche cir­cum­stances. Over­all it seems AI must progress slower if its suc­cess is driven by more dis­tinct ded­i­cated skills.
2. The ‘in­tel­li­gence am­plifi­ca­tion’ su­per­power seems much more im­por­tant than the oth­ers. It di­rectly leads to an in­tel­li­gence ex­plo­sion—a key rea­son we have seen so far to ex­pect any­thing ex­cit­ing to hap­pen with AI—while sev­eral oth­ers just al­low one-off grab­bing of re­sources (e.g. so­cial ma­nipu­la­tion and hack­ing). Note that this sug­gests an in­tel­li­gence ex­plo­sion could hap­pen with only this su­per­power, well be­fore an AI ap­peared to be hu­man-level.
3. Box 6 out­lines a spe­cific AI takeover sce­nario. A bunch of LessWrongers thought about other pos­si­bil­ities in this post.
4. Bostrom men­tions that so­cial ma­nipu­la­tion could al­low a ‘boxed’ AI to per­suade its gate­keep­ers to let it out. Some hu­mans have tried to demon­strate that this is a se­ri­ous haz­ard by simu­lat­ing the in­ter­ac­tion us­ing only an in­tel­li­gent hu­man in the place of the AI, in the ‘AI box ex­per­i­ment’. Ap­par­ently in both ‘offi­cial’ efforts the AI es­caped, though there have been other tri­als where the hu­man won.
5. How to mea­sure intelligence
Bostrom pointed to some efforts to de­sign more gen­eral in­tel­li­gence met­rics:
Legg: in­tel­li­gence is mea­sured in terms of re­ward in all re­ward-summable en­vi­ron­ments, weighted by com­plex­ity of the en­vi­ron­ment.
Hib­bard: in­tel­li­gence is mea­sured in terms of the hard­est en­vi­ron­ment you can pass, in a hi­er­ar­chy of in­creas­ingly hard environments
Dowe and Hernán­dez-Orallo have sev­eral pa­pers on the topic, and sum­ma­rize some other efforts. I haven’t looked at them enough to sum­ma­rize.
The Tur­ing Test is the most fa­mous test of ma­chine in­tel­li­gence. How­ever it only tests whether a ma­chine is at a spe­cific level so isn’t great for fine-grained mea­sure­ment of other lev­els of in­tel­li­gence. It is also of­ten mi­s­un­der­stood to mea­sure just whether a ma­chine can con­duct a nor­mal chat like a hu­man, rather than whether it can re­spond as ca­pa­bly as a hu­man to any­thing you can ask it.
For some spe­cific cog­ni­tive skills, there are other mea­sures already. e.g. ‘eco­nomic pro­duc­tivity’ can be mea­sured crudely in terms of prof­its made. Others seem like they could be de­vel­oped with­out too much difficulty. e.g. So­cial ma­nipu­la­tion could be mea­sured in terms of prob­a­bil­ities of suc­ceed­ing at ma­nipu­la­tion tasks—this test doesn’t ex­ist as far as I know, but it doesn’t seem pro­hibitively difficult to make.
6. Will we be able to colonize the stars?
Nick Beck­stead looked into it re­cently. Sum­mary: prob­a­bly.

In-depth investigations

If you are par­tic­u­larly in­ter­ested in these top­ics, and want to do fur­ther re­search, these are a few plau­si­ble di­rec­tions, al­most en­tirely taken from Luke Muehlhauser’s list, with­out my look­ing into them fur­ther.

  1. Try to de­velop met­rics for spe­cific im­por­tant cog­ni­tive abil­ities, in­clud­ing gen­eral in­tel­li­gence. Build on the ideas of Legg, Yud­kowsky, Go­ertzel, Her­nan­dez-Orallo & Dowe, etc.

  2. What is the con­struct val­idity of non-an­thro­po­mor­phic in­tel­li­gence mea­sures? In other words, are there con­ver­gently in­stru­men­tal pre­dic­tion and plan­ning al­gorithms? E.g. can one tend to get agents that are good at pre­dict­ing economies but not as­tro­nom­i­cal events? Or do self-mod­ify­ing agents in a com­pet­i­tive en­vi­ron­ment tend to con­verge to­ward a spe­cific sta­ble at­trac­tor in gen­eral in­tel­li­gence space?

  3. Sce­nario anal­y­sis: What are some con­crete AI paths to in­fluence over world af­fairs? See pro­ject guide here.

  4. How much of hu­man­ity’s cos­mic en­dow­ment can we plau­si­bly make pro­duc­tive use of given AGI? One way to ex­plore this ques­tion is via var­i­ous fol­low-ups to Arm­strong & Sand­berg (2013). Sand­berg lists sev­eral po­ten­tial fol­low-up stud­ies in this in­ter­view, for ex­am­ple (1) get more pre­cise mea­sure­ments of the dis­tri­bu­tion of large par­ti­cles in in­ter­stel­lar and in­ter­galac­tic space, and (2) an­a­lyze how well differ­ent long-term storable en­ergy sources scale. See Beck­stead (2014).

    If you are in­ter­ested in any­thing like this, you might want to men­tion it in the com­ments, and see whether other peo­ple have use­ful thoughts.

    How to proceed

    This has been a col­lec­tion of notes on the chap­ter. The most im­por­tant part of the read­ing group though is dis­cus­sion, which is in the com­ments sec­tion. I pose some ques­tions for you there, and I in­vite you to add your own. Please re­mem­ber that this group con­tains a va­ri­ety of lev­els of ex­per­tise: if a line of dis­cus­sion seems too ba­sic or too in­com­pre­hen­si­ble, look around for one that suits you bet­ter!

    Next week, we will talk about the or­thog­o­nal­ity of in­tel­li­gence and goals, sec­tion 9. To pre­pare, read The re­la­tion be­tween in­tel­li­gence and mo­ti­va­tion from Chap­ter 7. The dis­cus­sion will go live at 6pm Pa­cific time next Mon­day Novem­ber 10. Sign up to be no­tified here.