The Vul­ner­able World Hy­po­thesis (by Bostrom)

Link post

Nick Bostrom has put up a new work­ing pa­per to his per­sonal site (for the first time in two years?), called The Vul­ner­able World Hy­po­thesis.

I don’t think I have time to read it all, but I’d be in­ter­ested to see people com­ment with some choice quotes from the pa­per, and also read people’s opin­ions on the ideas within it.

To get the ba­sics, be­low I’ve writ­ten down the head­ings into a table of con­tents, copied in a few defin­i­tions I found when skim­ming, and also copied over the con­clu­sion (which seemed to me more read­able and use­ful than the ab­stract).


  • Is there a black ball in the urn of pos­sible in­ven­tions?

  • A thought ex­per­i­ment: easy nukes

  • The vul­ner­able world hypothesis

    • VWH: If tech­no­lo­gical de­vel­op­ment con­tin­ues then a set of cap­ab­il­it­ies will at some point be at­tained that make the dev­ast­a­tion of civil­iz­a­tion ex­tremely likely, un­less civil­iz­a­tion suf­fi­ciently exits the semi-an­archic de­fault con­di­tion.

  • Ty­po­logy of vulnerabilities

    • Type-1 (“easy nukes”)

      • Type-1 vul­ner­ab­il­ity: There is some tech­no­logy which is so de­struct­ive and so easy to use that, given the semi-an­archic de­fault con­di­tion, the ac­tions of act­ors in the apo­ca­lyptic re­sid­ual make civil­iz­a­tional dev­ast­a­tion ex­tremely likely.

    • Type-2a (“safe first strike”)

      • Type-2a vul­ner­ab­il­ity: There is some level of tech­no­logy at which power­ful act­ors have the abil­ity to pro­duce civil­iz­a­tion-dev­ast­at­ing harms and, in the semi-an­archic de­fault con­di­tion, face in­cent­ives to use that abil­ity.

    • Type-2b (“worse global warm­ing”)

      • Type-2b vul­ner­ab­il­ity: There is some level of tech­no­logy at which, in the semi-an­archic de­fault con­di­tion, a great many act­ors face in­cent­ives to take some slightly dam­aging ac­tion such that the com­bined ef­fect of those ac­tions is civil­iz­a­tional dev­ast­a­tion.

    • Type-0 (“sur­pris­ing strange­lets”)

      • Type-0 vul­ner­ab­il­ity: There is some tech­no­logy that car­ries a hid­den risk such that the de­fault out­come when it is dis­covered is in­ad­vert­ent civil­iz­a­tional dev­ast­a­tion. 47

  • Achiev­ing stabilization

    • Tech­no­lo­gical relinquishment

      • Prin­ciple of Dif­fer­en­tial Tech­no­lo­gical Devel­op­ment. Re­tard the de­vel­op­ment of dan­ger­ous and harm­ful tech­no­lo­gies, es­pe­cially ones that raise the level of ex­ist­en­tial risk; and ac­cel­er­ate the de­vel­op­ment of be­ne­fi­cial tech­no­lo­gies, es­pe­cially those that re­duce the ex­ist­en­tial risks posed by nature or by other tech­no­lo­gies.

    • Prefer­ence modification

    • Some spe­cific coun­ter­meas­ures and their limitations

    • Governance gaps

  • Pre­vent­ive policing

  • Global governance

  • Discussion

  • Conclusion


This pa­per has in­tro­duced a per­spect­ive from which we can more eas­ily see how civil­iz­a­tion is vul­ner­able to cer­tain types of pos­sible out­comes of our tech­no­lo­gical cre­ativ­ity—our draw­ing a meta­phor­ical black ball from the urn of in­ven­tions, which we have the power to ex­tract but not to put back in. We de­veloped a ty­po­logy of such po­ten­tial vul­ner­ab­il­it­ies, and showed how some of them res­ult from de­struc­tion be­com­ing too easy, oth­ers from per­ni­cious changes in the in­cent­ives fa­cing a few power­ful state act­ors or a large num­ber of weak act­ors.
We also ex­amined a vari­ety of pos­sible re­sponses and their lim­it­a­tions. We traced the root cause of our civil­iz­a­tional ex­pos­ure to two struc­tural prop­er­ties of the con­tem­por­ary world or­der: on the one hand, the lack of pre­vent­ive poli­cing ca­pa­city to block, with ex­tremely high re­li­ab­il­ity, in­di­vidu­als or small groups from car­ry­ing out ac­tions that are highly il­legal; and, on the other hand, the lack of global gov­ernance ca­pa­city to re­li­ably solve the gravest in­ter­na­tional co­ordin­a­tion prob­lems even when vi­tal na­tional in­terests by de­fault in­centiv­ize states to de­fect. Gen­eral sta­bil­iz­a­tion against po­ten­tial civil­iz­a­tional vul­ner­ab­il­it­ies—in a world where tech­no­lo­gical in­nov­a­tion is oc­cur­ring rap­idly along a wide fron­tier, and in which there are large num­bers of act­ors with a di­verse set of hu­man-re­cog­niz­able mo­tiv­a­tions—would re­quire that both of these gov­ernance gaps be elim­in­ated. Until such a time as this is ac­com­plished, hu­man­ity will re­main vul­ner­able to draw­ing a tech­no­lo­gical black ball.
Clearly, these re­flec­tions prove a pro tanto reason to sup­port strength­en­ing sur­veil­lance cap­ab­il­it­ies and pre­vent­ive poli­cing sys­tems and for fa­vor­ing a global gov­ernance re­gime that is cap­able of de­cis­ive ac­tion (whether based on uni­lat­eral he­ge­monic strength or power­ful mul­ti­lat­eral in­sti­tu­tions). However, we have not settled whether these things would be de­sir­able all-things-con­sidered, since do­ing so would re­quire ana­lyz­ing a num­ber of other strong con­sid­er­a­tions that lie out­side the scope of this pa­per.
Be­cause our main goal has been to put some sign­posts up in the mac­rostra­tegic land­scape, we have fo­cused our dis­cus­sion at a fairly ab­stract level, de­vel­op­ing con­cepts that can help us ori­ent ourselves (with re­spect to long-term out­comes and global de­sirab­il­it­ies) some­what in­de­pend­ently of the de­tails of our vary­ing local con­texts.
In prac­tice, were one to un­der­take an ef­fort to sta­bil­ize our civil­iz­a­tion against po­ten­tial black balls, one might find it prudent to fo­cus ini­tially on par­tial solu­tions and low-hanging fruits. Thus, rather than dir­ectly try­ing to bring about ex­tremely ef­fect­ive pre­vent­ive poli­cing or strong global gov­ernance, one might at­tempt to patch up par­tic­u­lar do­mains where black balls seem most likely to ap­pear. One could, for ex­ample, strengthen over­sight of bi­o­tech­no­logy-re­lated activ­it­ies by de­vel­op­ing bet­ter ways to track key ma­ter­i­als and equip­ment, and to mon­itor activ­it­ies within labs. One could also tighten know-your-cus­tomer reg­u­la­tions in the bi­otech sup­ply sec­tor, and ex­pand the use of back­ground checks for per­son­nel work­ing in cer­tain kinds of labs or in­volved with cer­tain kinds of ex­per­i­ment. One can im­prove whis­tleblower sys­tems, and try to raise bi­o­se­c­ur­ity stand­ards glob­ally. One could also pur­sue dif­fer­en­tial tech­no­lo­gical de­vel­op­ment, for in­stance by strength­en­ing the bio­lo­gical weapons con­ven­tion and main­tain­ing the global ta­boo on bio­lo­gical weapons. Fund­ing bod­ies and eth­ical ap­proval com­mit­tees could be en­cour­aged to take broader view of the po­ten­tial con­sequences of par­tic­u­lar lines of work, fo­cus­ing not only on risks to lab work­ers, test an­im­als, and hu­man re­search sub­jects, but also on ways that the hoped-for find­ings might lower the com­pet­ence bar for bi­o­ter­ror­ists down the road. Work that is pre­dom­in­antly pro­tect­ive (such as dis­ease out­break mon­it­or­ing, pub­lic health ca­pa­city build­ing, im­prove­ment of air fil­tra­tion devices) could be dif­fer­en­tially pro­moted.
Never­the­less, while pur­su­ing such lim­ited ob­ject­ives, one should bear in mind that the pro­tec­tion they would of­fer cov­ers only spe­cial sub­sets of scen­arios, and might be tem­por­ary. If one finds one­self in a po­s­i­tion to in­flu­ence the mac­ro­pa­ra­met­ers of pre­vent­ive poli­cing ca­pa­city or global gov­ernance ca­pa­city, one should con­sider that fun­da­mental changes in those do­mains may be the only way to achieve a gen­eral abil­ity to sta­bil­ize our civil­iz­a­tion against emer­ging tech­no­lo­gical vul­ner­ab­il­it­ies.