The Vulnerable World Hypothesis (by Bostrom)

Link post

Nick Bostrom has put up a new work­ing pa­per to his per­sonal site (for the first time in two years?), called The Vuln­er­a­ble World Hy­poth­e­sis.

I don’t think I have time to read it all, but I’d be in­ter­ested to see peo­ple com­ment with some choice quotes from the pa­per, and also read peo­ple’s opinions on the ideas within it.

To get the ba­sics, be­low I’ve writ­ten down the head­ings into a table of con­tents, copied in a few defi­ni­tions I found when skim­ming, and also copied over the con­clu­sion (which seemed to me more read­able and use­ful than the ab­stract).


  • Is there a black ball in the urn of pos­si­ble in­ven­tions?

  • A thought ex­per­i­ment: easy nukes

  • The vuln­er­a­ble world hypothesis

    • VWH: If tech­nolog­i­cal de­vel­op­ment con­tinues then a set of ca­pa­bil­ities will at some point be at­tained that make the dev­as­ta­tion of civ­i­liza­tion ex­tremely likely, un­less civ­i­liza­tion suffi­ciently ex­its the semi-an­ar­chic de­fault con­di­tion.

  • Ty­pol­ogy of vulnerabilities

    • Type-1 (“easy nukes”)

      • Type-1 vuln­er­a­bil­ity: There is some tech­nol­ogy which is so de­struc­tive and so easy to use that, given the semi-an­ar­chic de­fault con­di­tion, the ac­tions of ac­tors in the apoc­a­lyp­tic resi­d­ual make civ­i­liza­tional dev­as­ta­tion ex­tremely likely.

    • Type-2a (“safe first strike”)

      • Type-2a vuln­er­a­bil­ity: There is some level of tech­nol­ogy at which pow­er­ful ac­tors have the abil­ity to pro­duce civ­i­liza­tion-dev­as­tat­ing harms and, in the semi-an­ar­chic de­fault con­di­tion, face in­cen­tives to use that abil­ity.

    • Type-2b (“worse global warm­ing”)

      • Type-2b vuln­er­a­bil­ity: There is some level of tech­nol­ogy at which, in the semi-an­ar­chic de­fault con­di­tion, a great many ac­tors face in­cen­tives to take some slightly dam­ag­ing ac­tion such that the com­bined effect of those ac­tions is civ­i­liza­tional dev­as­ta­tion.

    • Type-0 (“sur­pris­ing strangelets”)

      • Type-0 vuln­er­a­bil­ity: There is some tech­nol­ogy that car­ries a hid­den risk such that the de­fault out­come when it is dis­cov­ered is in­ad­ver­tent civ­i­liza­tional dev­as­ta­tion. 47

  • Achiev­ing stabilization

    • Tech­nolog­i­cal relinquishment

      • Prin­ci­ple of Differ­en­tial Tech­nolog­i­cal Devel­op­ment. Re­tard the de­vel­op­ment of dan­ger­ous and harm­ful tech­nolo­gies, es­pe­cially ones that raise the level of ex­is­ten­tial risk; and ac­cel­er­ate the de­vel­op­ment of benefi­cial tech­nolo­gies, es­pe­cially those that re­duce the ex­is­ten­tial risks posed by na­ture or by other tech­nolo­gies.

    • Prefer­ence modification

    • Some spe­cific coun­ter­mea­sures and their limitations

    • Gover­nance gaps

  • Preven­tive policing

  • Global governance

  • Discussion

  • Conclusion


This pa­per has in­tro­duced a per­spec­tive from which we can more eas­ily see how civ­i­liza­tion is vuln­er­a­ble to cer­tain types of pos­si­ble out­comes of our tech­nolog­i­cal cre­ativity—our draw­ing a metaphor­i­cal black ball from the urn of in­ven­tions, which we have the power to ex­tract but not to put back in. We de­vel­oped a ty­pol­ogy of such po­ten­tial vuln­er­a­bil­ities, and showed how some of them re­sult from de­struc­tion be­com­ing too easy, oth­ers from per­ni­cious changes in the in­cen­tives fac­ing a few pow­er­ful state ac­tors or a large num­ber of weak ac­tors.
We also ex­am­ined a va­ri­ety of pos­si­ble re­sponses and their limi­ta­tions. We traced the root cause of our civ­i­liza­tional ex­po­sure to two struc­tural prop­er­ties of the con­tem­po­rary world or­der: on the one hand, the lack of pre­ven­tive polic­ing ca­pac­ity to block, with ex­tremely high re­li­a­bil­ity, in­di­vi­d­u­als or small groups from car­ry­ing out ac­tions that are highly ille­gal; and, on the other hand, the lack of global gov­er­nance ca­pac­ity to re­li­ably solve the gravest in­ter­na­tional co­or­di­na­tion prob­lems even when vi­tal na­tional in­ter­ests by de­fault in­cen­tivize states to defect. Gen­eral sta­bi­liza­tion against po­ten­tial civ­i­liza­tional vuln­er­a­bil­ities—in a world where tech­nolog­i­cal in­no­va­tion is oc­cur­ring rapidly along a wide fron­tier, and in which there are large num­bers of ac­tors with a di­verse set of hu­man-rec­og­niz­able mo­ti­va­tions—would re­quire that both of these gov­er­nance gaps be elimi­nated. Un­til such a time as this is ac­com­plished, hu­man­ity will re­main vuln­er­a­ble to draw­ing a tech­nolog­i­cal black ball.
Clearly, these re­flec­tions prove a pro tanto rea­son to sup­port strength­en­ing surveillance ca­pa­bil­ities and pre­ven­tive polic­ing sys­tems and for fa­vor­ing a global gov­er­nance regime that is ca­pa­ble of de­ci­sive ac­tion (whether based on unilat­eral hege­monic strength or pow­er­ful mul­ti­lat­eral in­sti­tu­tions). How­ever, we have not set­tled whether these things would be de­sir­able all-things-con­sid­ered, since do­ing so would re­quire an­a­lyz­ing a num­ber of other strong con­sid­er­a­tions that lie out­side the scope of this pa­per.
Be­cause our main goal has been to put some sign­posts up in the macros­trate­gic land­scape, we have fo­cused our dis­cus­sion at a fairly ab­stract level, de­vel­op­ing con­cepts that can help us ori­ent our­selves (with re­spect to long-term out­comes and global de­sir­a­bil­ities) some­what in­de­pen­dently of the de­tails of our vary­ing lo­cal con­texts.
In prac­tice, were one to un­der­take an effort to sta­bi­lize our civ­i­liza­tion against po­ten­tial black balls, one might find it pru­dent to fo­cus ini­tially on par­tial solu­tions and low-hang­ing fruits. Thus, rather than di­rectly try­ing to bring about ex­tremely effec­tive pre­ven­tive polic­ing or strong global gov­er­nance, one might at­tempt to patch up par­tic­u­lar do­mains where black balls seem most likely to ap­pear. One could, for ex­am­ple, strengthen over­sight of biotech­nol­ogy-re­lated ac­tivi­ties by de­vel­op­ing bet­ter ways to track key ma­te­ri­als and equip­ment, and to mon­i­tor ac­tivi­ties within labs. One could also tighten know-your-cus­tomer reg­u­la­tions in the biotech sup­ply sec­tor, and ex­pand the use of back­ground checks for per­son­nel work­ing in cer­tain kinds of labs or in­volved with cer­tain kinds of ex­per­i­ment. One can im­prove whistle­blower sys­tems, and try to raise biose­cu­rity stan­dards globally. One could also pur­sue differ­en­tial tech­nolog­i­cal de­vel­op­ment, for in­stance by strength­en­ing the biolog­i­cal weapons con­ven­tion and main­tain­ing the global taboo on biolog­i­cal weapons. Fund­ing bod­ies and eth­i­cal ap­proval com­mit­tees could be en­couraged to take broader view of the po­ten­tial con­se­quences of par­tic­u­lar lines of work, fo­cus­ing not only on risks to lab work­ers, test an­i­mals, and hu­man re­search sub­jects, but also on ways that the hoped-for find­ings might lower the com­pe­tence bar for bioter­ror­ists down the road. Work that is pre­dom­i­nantly pro­tec­tive (such as dis­ease out­break mon­i­tor­ing, pub­lic health ca­pac­ity build­ing, im­prove­ment of air fil­tra­tion de­vices) could be differ­en­tially pro­moted.
Nev­er­the­less, while pur­su­ing such limited ob­jec­tives, one should bear in mind that the pro­tec­tion they would offer cov­ers only spe­cial sub­sets of sce­nar­ios, and might be tem­po­rary. If one finds one­self in a po­si­tion to in­fluence the macropa­ram­e­ters of pre­ven­tive polic­ing ca­pac­ity or global gov­er­nance ca­pac­ity, one should con­sider that fun­da­men­tal changes in those do­mains may be the only way to achieve a gen­eral abil­ity to sta­bi­lize our civ­i­liza­tion against emerg­ing tech­nolog­i­cal vuln­er­a­bil­ities.