Superintelligence 28: Collaboration

This is part of a weekly read­ing group on Nick Bostrom’s book, Su­per­in­tel­li­gence. For more in­for­ma­tion about the group, and an in­dex of posts so far see the an­nounce­ment post. For the sched­ule of fu­ture top­ics, see MIRI’s read­ing guide.

Wel­come. This week we dis­cuss the twenty-eighth sec­tion in the read­ing guide: Col­lab­o­ra­tion.

This post sum­ma­rizes the sec­tion, and offers a few rele­vant notes, and ideas for fur­ther in­ves­ti­ga­tion. Some of my own thoughts and ques­tions for dis­cus­sion are in the com­ments.

There is no need to pro­ceed in or­der through this post, or to look at ev­ery­thing. Feel free to jump straight to the dis­cus­sion. Where ap­pli­ca­ble and I re­mem­ber, page num­bers in­di­cate the rough part of the chap­ter that is most re­lated (not nec­es­sar­ily that the chap­ter is be­ing cited for the spe­cific claim).

Read­ing: “Col­lab­o­ra­tion” from Chap­ter 14


  1. The de­gree of col­lab­o­ra­tion among those build­ing AI might af­fect the out­come a lot. (p246)

  2. If mul­ti­ple pro­jects are close to de­vel­op­ing AI, and the first will reap sub­stan­tial benefits, there might be a ‘race dy­namic’ where safety is sac­ri­ficed on all sides for a greater chance of win­ning. (247-8)

  3. Avert­ing such a race dy­namic with col­lab­o­ra­tion should have these benefits:

    1. More safety

    2. Slower AI progress (al­low­ing more con­sid­ered re­sponses)

    3. Less other dam­age from con­flict over the race

    4. More shar­ing of ideas for safety

    5. More equitable out­comes (for a va­ri­ety of rea­sons)

  4. Equitable out­comes are good for var­i­ous moral and pru­den­tial rea­sons. They may also be eas­ier to com­pro­mise over than ex­pected, be­cause hu­mans have diminish­ing re­turns to re­sources. How­ever in the fu­ture, their re­turns may be less diminish­ing (e.g. if re­sources can buy more time in­stead of en­ter­tain­ments one has no time for).

  5. Col­lab­o­ra­tion be­fore a tran­si­tion to an AI econ­omy might af­fect how much col­lab­o­ra­tion there is af­ter­wards. This might not be straight­for­ward. For in­stance, if a sin­gle­ton is the de­fault out­come, then low col­lab­o­ra­tion be­fore a tran­si­tion might lead to a sin­gle­ton (i.e. high col­lab­o­ra­tion) af­ter­wards, and vice versa. (p252)

  6. An in­ter­na­tional col­lab­o­ra­tive AI pro­ject might de­serve nearly in­fea­si­ble lev­els of se­cu­rity, such as be­ing al­most com­pletely iso­lated from the world. (p253)

  7. It is good to start col­lab­o­ra­tion early, to benefit from be­ing ig­no­rant about who will benefit more from it, but hard be­cause the pro­ject is not yet rec­og­nized as im­por­tant. Per­haps the ap­pro­pri­ate col­lab­o­ra­tion at this point is to pro­pound some­thing like ‘the com­mon good prin­ci­ple’. (p253)

  8. ‘The com­mon good prin­ci­ple’: Su­per­in­tel­li­gence should be de­vel­oped only for the benefit of all of hu­man­ity and in the ser­vice of widely shared eth­i­cal ideals. (p254)

Another view

Miles Brundage on the Col­lab­o­ra­tion sec­tion:

This is an im­por­tant topic, and Bostrom says many things I agree with. A few places where I think the is­sues are less clear:

  • Many of Bostrom’s pro­pos­als de­pend on AI re­calc­i­trance be­ing low. For in­stance, a highly se­cre­tive in­ter­na­tional effort makes less sense if build­ing AI is a long and in­cre­men­tal slog. Re­calc­i­trance may well be low, but this isn’t ob­vi­ous, and it is good to rec­og­nize this de­pen­dency and con­sider what pro­pos­als would be ap­pro­pri­ate for other re­calc­i­trance lev­els.

  • Arms races are ubiquitous in our global cap­i­tal­ist econ­omy, and AI is already in one. Arms races can stem from mar­ket com­pe­ti­tion by firms or state-driven na­tional se­cu­rity-ori­ented R+D efforts as well as com­plex com­bi­na­tions of these, sug­gest­ing the need for fur­ther re­search on the re­la­tion­ship be­tween AI de­vel­op­ment, na­tional se­cu­rity, and global cap­i­tal­ist mar­ket dy­nam­ics. It’s un­clear how well the sim­ple arms race model here matches the re­al­ity of the cur­rent AI arms race or fu­ture vari­a­tions of it. The model’s main value is prob­a­bly in prob­ing as­sump­tions and in­spiring the de­vel­op­ment of richer mod­els, as it’s prob­a­bly too sim­ple in to fit re­al­ity well as-is. For in­stance, it is un­clear that safety and ca­pa­bil­ity are close to or­thog­o­nal in prac­tice to­day. If many AI peo­ple gen­uinely care about safety (which the quan­tity and qual­ity of sig­na­to­ries to the FLI open let­ter sug­gests is plau­si­ble), or work on eco­nom­i­cally rele­vant near-term safety is­sues at each point is im­por­tant, or con­sumers re­ward eth­i­cal com­pa­nies with their pur­chases, then bet­ter AI firms might in­vest a lot in safety for self-in­ter­ested as well as al­tru­is­tic rea­sons. Also, if the AI field shifts to fo­cus more on hu­man-com­ple­men­tary in­tel­li­gence that re­quires and benefits from long-term, high-fre­quency in­ter­ac­tion with hu­mans, then safety and ca­pa­bil­ity may be syn­er­gis­tic rather than trad­ing off against each other. In­cen­tives re­lated to re­search pri­ori­ties should also be con­sid­ered in a strate­gic anal­y­sis of AI gov­er­nance (e.g. are AI re­searchers cur­rently in­cen­tivized only to demon­strate ca­pa­bil­ity ad­vances in the pa­pers they write, and could in­cen­tives be changed or the aims and scope of the field re­defined so that more progress is made on safety is­sues?).

  • ‘AI’ is too course grained a unit for a strate­gic anal­y­sis of col­lab­o­ra­tion. The na­ture and ur­gency of col­lab­o­ra­tion de­pends on the de­tails of what is be­ing de­vel­oped. An enor­mous va­ri­ety of ar­tifi­cial in­tel­li­gence re­search is pos­si­ble and the goals of the field are un­der­con­strained by na­ture (e.g. we can model sys­tems based on ap­prox­i­ma­tions of ra­tio­nal­ity, or on hu­mans, or an­i­mals, or some­thing else en­tirely, based on cu­ri­os­ity, so­cial im­pact, and other con­sid­er­a­tions that could be more ex­plic­itly eval­u­ated), and are thus open to change in the fu­ture. We need to think more about differ­en­tial tech­nol­ogy de­vel­op­ment within the do­main of AI. This too will af­fect the ur­gency and na­ture of co­op­er­a­tion.


1. In Bostrom’s de­scrip­tion of his model, it is a bit un­clear how safety pre­cau­tions af­fect perfor­mance. He says ‘one can model each team’s perfor­mance as a func­tion of its ca­pa­bil­ity (mea­sur­ing its raw abil­ity and luck) and a penalty term cor­re­spond­ing to the cost of its safety pre­cau­tions’ (p247), which sounds like they are purely a nega­tive. How­ever this wouldn’t make sense: if safety pre­cau­tions were just a cost, then re­gard­less of com­pe­ti­tion, no­body would in­vest in safety. In re­al­ity, who­ever wins con­trol over the world benefits a lot from what­ever safety pre­cau­tions have been taken. If the world is de­stroyed in the pro­cess of an AI tran­si­tion, they have lost ev­ery­thing! I think this is the model Bostrom means to re­fer to. While he says it may lead to min­i­mum pre­cau­tions, note that in many mod­els it would merely lead to less safety than one would want. If you are spend­ing noth­ing on safety, and thus go­ing to take over a world that is worth noth­ing, you would of­ten pre­fer to move to a lower prob­a­bil­ity of win­ning a more valuable world. Arm­strong, Bostrom and Shul­man dis­cuss this kind of model in more depth.

2. If you are in­ter­ested in the game the­ory of con­flicts like this, The Strat­egy of Con­flict is a great book.

3. Given the gains to com­peti­tors co­op­er­at­ing to not de­stroy the world that they are try­ing to take over, re­search on how to ar­range co­op­er­a­tion seems helpful for all sides. The situ­a­tion is much like a tragedy of the com­mons, ex­cept for the win­ner-takes-all as­pect: each per­son gains from ne­glect­ing safety, while ex­ert­ing a small cost on ev­ery­one. Academia seems to be pretty in­ter­ested in re­solv­ing tragedies of the com­mons, so per­haps that liter­a­ture is worth try­ing to ap­ply here.

4. The most fa­mous arms race is ar­guably the nu­clear one. I won­der to what ex­tent this was a ma­jor arms race be­cause nu­clear weapons were des­tined to be an un­usu­ally mas­sive jump in progress. If this was im­por­tant, it leads to the ques­tion of whether we have rea­son to ex­pect any­thing similar in AI.

In-depth investigations

If you are par­tic­u­larly in­ter­ested in these top­ics, and want to do fur­ther re­search, these are a few plau­si­ble di­rec­tions, some in­spired by Luke Muehlhauser’s list, which con­tains many sug­ges­tions re­lated to parts of Su­per­in­tel­li­gence. Th­ese pro­jects could be at­tempted at var­i­ous lev­els of depth.

  1. Ex­plore other mod­els of com­pet­i­tive AI de­vel­op­ment.

  2. What policy in­ter­ven­tions help in pro­mot­ing col­lab­o­ra­tion?

  3. What kinds of situ­a­tions pro­duce arms races?

  4. Ex­am­ine in­ter­na­tional col­lab­o­ra­tion on ma­jor in­no­va­tive tech­nol­ogy. How of­ten does it hap­pen? What blocks it from hap­pen­ing more? What are the nec­es­sary con­di­tions? Ex­am­ples: Con­cord jet, LHC, in­ter­na­tional space sta­tion, etc.

  5. Con­duct a broad sur­vey of past and cur­rent civ­i­liza­tional com­pe­tence. In what ways, and un­der what con­di­tions, do hu­man civ­i­liza­tions show com­pe­tence vs. in­com­pe­tence? Which kinds of prob­lems do they han­dle well or poorly? Similar in scope and am­bi­tion to, say, Per­row’s Nor­mal Ac­ci­dents and Sa­gan’s The Limits of Safety. The aim is to get some in­sight into the like­li­hood of our civ­i­liza­tion han­dling var­i­ous as­pects of the su­per­in­tel­li­gence challenge well or poorly. Some ini­tial steps were taken here and here.

  6. What hap­pens when gov­ern­ments ban or re­strict cer­tain kinds of tech­nolog­i­cal de­vel­op­ment? What hap­pens when a cer­tain kind of tech­nolog­i­cal de­vel­op­ment is banned or re­stricted in one coun­try but not in other coun­tries where tech­nolog­i­cal de­vel­op­ment sees heavy in­vest­ment?

  7. What kinds of in­no­va­tive tech­nol­ogy pro­jects do gov­ern­ments mon­i­tor, shut down, or na­tion­al­ize? How likely are ma­jor gov­ern­ments to mon­i­tor, shut down, or na­tion­al­ize se­ri­ous AGI pro­jects?

  8. How likely is it that AGI will be a sur­prise to most policy-mak­ers and in­dus­try lead­ers? How much ad­vance warn­ing are they likely to have? Some notes on this here.

    If you are in­ter­ested in any­thing like this, you might want to men­tion it in the com­ments, and see whether other peo­ple have use­ful thoughts.

    How to proceed

    This has been a col­lec­tion of notes on the chap­ter. The most im­por­tant part of the read­ing group though is dis­cus­sion, which is in the com­ments sec­tion. I pose some ques­tions for you there, and I in­vite you to add your own. Please re­mem­ber that this group con­tains a va­ri­ety of lev­els of ex­per­tise: if a line of dis­cus­sion seems too ba­sic or too in­com­pre­hen­si­ble, look around for one that suits you bet­ter!

    Next week, we will talk about what to do in this ‘crunch time’. To pre­pare, read Chap­ter 15. The dis­cus­sion will go live at 6pm Pa­cific time next Mon­day 30 March. Sign up to be no­tified here.