Superintelligence 26: Science and technology strategy

This is part of a weekly read­ing group on Nick Bostrom’s book, Su­per­in­tel­li­gence. For more in­for­ma­tion about the group, and an in­dex of posts so far see the an­nounce­ment post. For the sched­ule of fu­ture top­ics, see MIRI’s read­ing guide.

Wel­come. This week we dis­cuss the twenty-sixth sec­tion in the read­ing guide: Science and tech­nol­ogy strat­egy. Sorry for post­ing late—my car broke.

This post sum­ma­rizes the sec­tion, and offers a few rele­vant notes, and ideas for fur­ther in­ves­ti­ga­tion. Some of my own thoughts and ques­tions for dis­cus­sion are in the com­ments.

There is no need to pro­ceed in or­der through this post, or to look at ev­ery­thing. Feel free to jump straight to the dis­cus­sion. Where ap­pli­ca­ble and I re­mem­ber, page num­bers in­di­cate the rough part of the chap­ter that is most re­lated (not nec­es­sar­ily that the chap­ter is be­ing cited for the spe­cific claim).

Read­ing: “Science and tech­nol­ogy strat­egy” from Chap­ter 14


  1. This sec­tion will in­tro­duce con­cepts that are use­ful for think­ing about long term is­sues in sci­ence and tech­nol­ogy (p228)

  2. Per­son af­fect­ing per­spec­tive: one should act in the best in­ter­ests of ev­ery­one who already ex­ists, or who will ex­ist in­de­pen­dent of one’s choices (p228)

  3. Im­per­sonal per­spec­tive: one should act in the best in­ter­ests of ev­ery­one, in­clud­ing those who may be brought into ex­is­tence by one’s choices. (p228)

  4. Tech­nolog­i­cal com­ple­tion con­jec­ture: “If sci­en­tific and tech­nolog­i­cal de­vel­op­ment efforts do not cease, then all im­por­tant ba­sic ca­pa­bil­ities that could be ob­tained through some pos­si­ble tech­nol­ogy will be ob­tained.” (p229)

    1. This does not im­ply that it is fu­tile to try to steer tech­nol­ogy. Efforts may cease. It might also mat­ter ex­actly when things are de­vel­oped, who de­vel­ops them, and in what con­text.

  5. Prin­ci­ple of differ­en­tial tech­nolog­i­cal de­vel­op­ment: one should slow the de­vel­op­ment of dan­ger­ous and harm­ful tech­nolo­gies rel­a­tive to benefi­cial tech­nolo­gies (p230)

  6. We have a preferred or­der for some tech­nolo­gies, e.g. it is bet­ter to have su­per­in­tel­li­gence later rel­a­tive to so­cial progress, but ear­lier rel­a­tive to other ex­is­ten­tial risks. (p230-233)

  7. If a macrostruc­tural de­vel­op­ment ac­cel­er­a­tor is a magic lever which slows the large scale fea­tures of his­tory (e.g. tech­nolog­i­cal change, geopoli­ti­cal dy­nam­ics) while leav­ing the small scale fea­tures the same, then we can ask whether pul­ling the lever would be a good idea (p233). The main way Bostrom con­cludes that it mat­ters is by af­fect­ing how well pre­pared hu­man­ity is for fu­ture tran­si­tions.

  8. State risk: a risk that per­sists while you are in a cer­tain situ­a­tion, such that the amount of risk is a func­tion of the time spent there. e.g. risk from as­ter­oids, while we don’t have tech­nol­ogy to redi­rect them. (p233-4)

  9. Step risk: a risk aris­ing from a tran­si­tion. Here the amount of risk is mostly not a func­tion of how long the tran­si­tion takes. e.g. travers­ing a minefield: this is not es­pe­cially safer if you run faster. (p234)

  10. Tech­nol­ogy cou­pling: a pre­dictable timing re­la­tion­ship be­tween two tech­nolo­gies, such that has­ten­ing of the first tech­nol­ogy will has­ten the sec­ond, ei­ther be­cause the sec­ond is a pre­cur­sor or be­cause it is a nat­u­ral con­se­quence. (p236-8) e.g. brain em­u­la­tion is plau­si­bly cou­pled to ‘neu­ro­mor­phic’ AI, be­cause the un­der­stand­ing re­quired to em­u­late a brain might al­low one to more quickly cre­ate an AI on similar prin­ci­ples.

  11. Se­cond guess­ing: act­ing as if “by treat­ing oth­ers as ir­ra­tional and play­ing to their bi­ases and mis­con­cep­tions it is pos­si­ble to elicit a re­sponse from them that is more com­pe­tent than if a case had been pre­sented hon­estly and forthrightly to their ra­tio­nal fac­ul­ties” (p238-40)

Another view

There is a com­mon view which says we should not act on de­tailed ab­stract ar­gu­ments about the far fu­ture like those of this sec­tion. Here Holden Karnofsky ex­em­plifies it:

I have of­ten been challenged to ex­plain how one could pos­si­bly rec­on­cile (a) car­ing a great deal about the far fu­ture with (b) donat­ing to one of GiveWell’s top char­i­ties. My gen­eral re­sponse is that in the face of suffi­cient un­cer­tainty about one’s op­tions, and lack of con­vic­tion that there are good (in the sense of high ex­pected value) op­por­tu­ni­ties to make an enor­mous differ­ence, it is ra­tio­nal to try to make a smaller but ro­bustly pos­i­tived­iffer­ence, whether or not one can trace a spe­cific causal path­way from do­ing this small amount of good to mak­ing a large im­pact on the far fu­ture. A few brief ar­gu­ments in sup­port of this po­si­tion:

  • I be­lieve that the track record of “tak­ing ro­bustly strong op­por­tu­ni­ties to do ‘some­thing good’” is far bet­ter than the track record of “tak­ing ac­tions whose value is con­tin­gent on high-un­cer­tainty ar­gu­ments about where the high­est util­ity lies, and/​or ar­gu­ments about what is likely to hap­pen in the far fu­ture.” This is true even when one eval­u­ates track record only in terms of seem­ing im­pact on the far fu­ture. The de­vel­op­ments that seem most pos­i­tive in ret­ro­spect – from large ones like the de­vel­op­ment of the steam en­g­ine to small ones like the many eco­nomic con­tri­bu­tions that fa­cil­i­tated strong over­all growth – seem to have been driven by the former ap­proach, and I’m not aware of many ex­am­ples in which the lat­ter ap­proach has yielded great benefits.

  • I see some sense in which the world’s over­all civ­i­liza­tional ecosys­tem seems to have done a bet­ter job op­ti­miz­ing for the far fu­ture than any of the world’s in­di­vi­d­ual minds. It’s of­ten the case that peo­ple act­ing on rel­a­tively short-term, tan­gible con­sid­er­a­tions (es­pe­cially when they did so with cre­ativity, in­tegrity, trans­parency, con­sen­su­al­ity, and pur­suit of gain via value cre­ation rather than value trans­fer) have done good in ways they them­selves wouldn’t have been able to fore­see. If this is cor­rect, it seems to im­ply that one should be fo­cused on “play­ing one’s role as well as pos­si­ble” – on find­ing op­por­tu­ni­ties to “beat the broad mar­ket” (to do more good than peo­ple with similar goals would be able to) rather than pour­ing one’s re­sources into the ar­eas that non-ro­bust es­ti­mates have in­di­cated as most im­por­tant to the far fu­ture.

  • The pro­cess of try­ing to ac­com­plish tan­gible good can lead to a great deal of learn­ing and un­ex­pected pos­i­tive de­vel­op­ments, more so (in my view) than the pro­cess of putting re­sources into a low-feed­back en­deavor based on one’s cur­rent best-guess the­ory. In my con­ver­sa­tion with Luke and Eliezer, the two of them hy­poth­e­sized that the great­est pos­i­tive benefit of sup­port­ing GiveWell’s top char­i­ties may have been to raise the pro­file, in­fluence, and learn­ing abil­ities of GiveWell. If this were true, I don’t be­lieve it would be an in­ex­pli­ca­ble stroke of luck for donors to top char­i­ties; rather, it would be the sort of de­vel­op­ment (fa­cil­i­tat­ing feed­back loops that lead to learn­ing, or­ga­ni­za­tional de­vel­op­ment, grow­ing in­fluence, etc.) that is of­ten as­so­ci­ated with “do­ing some­thing well” as op­posed to “do­ing the most worth­while thing poorly.”

  • I see mul­ti­ple rea­sons to be­lieve that con­tribut­ing to gen­eral hu­man em­pow­er­ment miti­gates global catas­trophic risks. I laid some of these out in a blog post and dis­cussed them fur­ther in my con­ver­sa­tion with Luke and Eliezer.


1. Tech­nolog­i­cal com­ple­tion timelines game
The tech­nolog­i­cal com­ple­tion con­jec­ture says that all the ba­sic tech­nolog­i­cal ca­pa­bil­ities will even­tu­ally be de­vel­oped. But when is ‘even­tu­ally’, usu­ally? Do things get de­vel­oped ba­si­cally as soon as de­vel­op­ing them is not pro­hibitively ex­pen­sive, or is think­ing of the thing of­ten a bot­tle­neck? This is rele­vant to how much we can hope to in­fluence the timing of tech­nolog­i­cal de­vel­op­ments.

Here is a fun game: How many things can you find that could have been prof­itably de­vel­oped much ear­lier than they were?

Some start­ing sug­ges­tions, which I haven’t looked into:

Wheeled lug­gage: in­vented in the 1970s, though hu­man­ity had had both wheels and lug­gage for a while.

Hot air bal­loons: fly­ing pa­per lan­terns us­ing the same prin­ci­ple were ap­par­ently used be­fore 200AD, while a manned bal­loon wasn’t used un­til 1783.

Peni­cillin: mould was ap­par­ently tra­di­tion­ally used for an­tibac­te­rial prop­er­ties in sev­eral cul­tures, but lots of things are tra­di­tion­ally used for lots of things. By the 1870s many sci­en­tists had noted that spe­cific moulds in­hibited bac­te­rial growth.

Wheels: Early toys from the Amer­i­cas ap­pear to have had wheels (here and pic­tured is one from 1-900AD; Wikipe­dia claims such toys were around as early as 1500BC). How­ever wheels were ap­par­ently not used for more sub­stan­tial trans­port in the Amer­i­cas un­til much later.

Image: “Re­mo­jadas Wheeled Figurine”

There are also cases where hu­man­ity has for­got­ten im­por­tant in­sights, and then re­dis­cov­ered them again much later, which sug­gests strongly that they could have been de­vel­oped ear­lier.

2. How does eco­nomic growth af­fect AI risk?

Eliezer Yud­kowsky ar­gues that eco­nomic growth in­creases risk. I ar­gue that he has the sign wrong. Others ar­gue that prob­a­bly lots of other fac­tors mat­ter more any­way. Luke Muehlhauser ex­pects that cog­ni­tive en­hance­ment is bad, largely based on Eliezer’s afore­men­tioned claim. He also points out that smarter peo­ple are differ­ent from more ra­tio­nal peo­ple. Paul Chris­ti­ano out­lines his own eval­u­a­tion of eco­nomic growth in gen­eral, on hu­man­ity’s long run welfare. He also dis­cusses the value of con­tinued tech­nolog­i­cal, eco­nomic and so­cial progress more com­pre­hen­si­bly here.

3. The per­son af­fect­ing perspective

Some in­ter­est­ing cri­tiques: the non-iden­tity prob­lem, tak­ing ad­di­tional peo­ple to be neu­tral makes other good or bad things neu­tral too, if you try to be con­sis­tent in nat­u­ral ways.

In-depth investigations

If you are par­tic­u­larly in­ter­ested in these top­ics, and want to do fur­ther re­search, these are a few plau­si­ble di­rec­tions, some in­spired by Luke Muehlhauser’s list, which con­tains many sug­ges­tions re­lated to parts of Su­per­in­tel­li­gence. Th­ese pro­jects could be at­tempted at var­i­ous lev­els of depth.

  1. Is macro-struc­tural ac­cel­er­a­tion good or bad on net for AI safety?

  2. Choose a par­tic­u­lar an­ti­ci­pated tech­nol­ogy. Is it’s de­vel­op­ment good or bad for AI safety on net?

  3. What is the over­all cur­rent level of “state risk” from ex­is­ten­tial threats?

  4. What are the ma­jor ex­is­ten­tial-threat “step risks” ahead of us, be­sides those from su­per­in­tel­li­gence?

  5. What are some ad­di­tional “tech­nol­ogy cou­plings,” in ad­di­tion to those named in Su­per­in­tel­li­gence, ch. 14?

  6. What are fur­ther preferred or­der­ings for tech­nolo­gies not men­tioned in this sec­tion?

If you are in­ter­ested in any­thing like this, you might want to men­tion it in the com­ments, and see whether other peo­ple have use­ful thoughts.

How to proceed

This has been a col­lec­tion of notes on the chap­ter. The most im­por­tant part of the read­ing group though is dis­cus­sion, which is in the com­ments sec­tion. I pose some ques­tions for you there, and I in­vite you to add your own. Please re­mem­ber that this group con­tains a va­ri­ety of lev­els of ex­per­tise: if a line of dis­cus­sion seems too ba­sic or too in­com­pre­hen­si­ble, look around for one that suits you bet­ter!

Next week, we will talk about the de­sir­a­bil­ity of hard­ware progress, and progress to­ward brain em­u­la­tion. To pre­pare, read “Path­ways and en­ablers” from Chap­ter 14. The dis­cus­sion will go live at 6pm Pa­cific time next Mon­day 16th March. Sign up to be no­tified here.