Soft takeoff can still lead to decisive strategic advantage

[Epistemic sta­tus: Ar­gu­ment by anal­ogy to his­tor­i­cal cases. Best case sce­nario it’s just one ar­gu­ment among many. Edit: Also, thanks to feed­back from oth­ers, es­pe­cially Paul, I in­tend to write a sig­nifi­cantly im­proved ver­sion of this post in the next two weeks.]

I have on sev­eral oc­ca­sions heard peo­ple say things like this:

The origi­nal Bostrom/​Yud­kowsky paradigm en­vi­sioned a sin­gle AI built by a sin­gle AI pro­ject, un­der­go­ing in­tel­li­gence ex­plo­sion all by it­self and at­tain­ing a de­ci­sive strate­gic ad­van­tage as a re­sult. How­ever, this is very un­re­al­is­tic. Dis­con­tin­u­ous jumps in tech­nolog­i­cal ca­pa­bil­ity are very rare, and it is very im­plau­si­ble that one pro­ject could pro­duce more in­no­va­tions than the rest of the world com­bined. In­stead we should ex­pect some­thing more like the In­dus­trial Revolu­tion: Con­tin­u­ous growth, spread among many pro­jects and fac­tions, shared via a com­bi­na­tion of trade and tech­nol­ogy steal­ing. We should not ex­pect any one pro­ject or AI to at­tain a de­ci­sive strate­gic ad­van­tage, be­cause there will always be other pro­jects and other AI that are only slightly less pow­er­ful, and coal­i­tions will act to coun­ter­bal­ance the tech­nolog­i­cal ad­van­tage of the fron­trun­ner. (para­phrased)

Pro­po­nents of this view of­ten cite Paul Chris­ti­ano in sup­port. Last week I heard him say he thinks the fu­ture will be “like the In­dus­trial Revolu­tion but 10x-100x faster.”

In this post, I as­sume that Paul’s slo­gan for the fu­ture is cor­rect and then nev­er­the­less push back against the view above. Ba­si­cally, I will ar­gue that even if the fu­ture is like the in­dus­trial rev­olu­tion only 10x-100x faster, there is a 30%+ chance that it will in­volve a sin­gle AI pro­ject (or a sin­gle AI) with the abil­ity to gain a de­ci­sive strate­gic ad­van­tage, if they so choose. (Whether or not they ex­er­cise that abil­ity is an­other mat­ter.)

Why am I in­ter­ested in this? Do I ex­pect some hu­man group to take over the world? No; in­stead what I think is that (1) an un­al­igned AI in the lead­ing pro­ject might take over the world, and (2) A hu­man pro­ject that suc­cess­fully al­igns their AI might re­frain from tak­ing over the world even if they have the abil­ity to do so, and in­stead use their ca­pa­bil­ities to e.g. help the United Na­tions en­force a ban on unau­tho­rized AGI pro­jects.

Na­tional ELO rat­ings dur­ing the in­dus­trial rev­olu­tion and the mod­ern era

In chess (and some other games) ELO rank­ings are used to com­pare play­ers. An av­er­age club player might be rank 1500; the world chess cham­pion might be 2800; com­puter chess pro­grams are even bet­ter. If one player has 400 points more than an­other, it means the first player would win with ~90% prob­a­bil­ity.

We could ap­ply this sys­tem to com­pare the war­mak­ing abil­ities of na­tion-states and coal­i­tions of na­tion-states. For ex­am­ple, in 1941 per­haps we could say that the ELO rank of the Axis pow­ers was ~300 points lower than the ELO rank of the rest of the world com­bined (be­cause what in fact hap­pened was the rest of the world com­bin­ing to defeat them, but it wasn’t a guaran­teed vic­tory). We could add that in 1939 the ELO rank of Ger­many was ~400 points higher than that of Poland, and that the ELO rank of Poland was prob­a­bly 400+ points higher than that of Lux­em­bourg.

We could make cross-tem­po­ral fan­tasy com­par­i­sons too. The ELO rank­ing of Ger­many in 1939 was prob­a­bly ~400 points greater than that of the en­tire world circa 1910, for ex­am­ple. (Vi­su­al­ize the en­tirety of 1939 Ger­many tele­port­ing back in time to 1910, and then imag­ine the havoc it would wreak.)

Claim 1A: If we were to es­ti­mate the ELO rank­ings of all na­tion-states and sets of na­tion-states (po­ten­tial al­li­ances) over the last 300 years, the rank of the most pow­er­ful na­tion-state at at a given year would on sev­eral oc­ca­sions be 400+ points greater than the rank of the en­tire world com­bined 30 years prior.

Claim 1B: Over the last 300 years there have been sev­eral oc­ca­sions in which one na­tion-state had the ca­pa­bil­ity to take over the en­tire world of 30 years prior.

I’m no his­to­rian, but I feel fairly con­fi­dent in these claims.

  • In naval his­tory, the best fleets in the world in 1850 were ob­so­lete by 1860 thanks to the in­tro­duc­tion of iron-hul­led steamships, and said steamships were them­selves ob­so­lete a decade or so later, and then those ships were ob­so­leted by the Dread­nought, and so on… This pro­cess con­tinued into the mod­ern era. By “Ob­so­leted” I mean some­thing like “A sin­gle ship of the new type could defeat the en­tire com­bined fleet of ves­sels of the old type.”

  • A similar story could be told about air power. In a dogfight be­tween planes of year 19XX and year 19XX+30, the sec­ond group of planes will be limited only by how much am­mu­ni­tion they can carry.

  • Small tech­nolog­i­cally ad­vanced na­tions have reg­u­larly beaten huge sprawl­ing em­pires and coal­i­tions. (See: Colo­nial­ism)

  • The en­tire world has been ba­si­cally carved up be­tween the small hand­ful of most-tech­nolog­i­cally ad­vanced na­tions for two cen­turies now. For ex­am­ple, any of the Great Pow­ers of 1910 (plus the USA) could have taken over all of Africa, Asia, South Amer­ica, etc. if not for the re­sis­tance that the other great pow­ers would put up. The same was true 40 years later and 40 years ear­lier.

I con­clude from this that if some great power in the era kicked off by the in­dus­trial rev­olu­tion had man­aged to “pull ahead” of the rest of the world more effec­tively than it ac­tu­ally did--30 years more effec­tively, in par­tic­u­lar—it re­ally would have been able to take over the world.

Claim 2: If the fu­ture is like the In­dus­trial Revolu­tion but 10x-100x faster, then cor­re­spond­ingly the tech­nolog­i­cal and eco­nomic power granted by be­ing 3 − 0.3 years ahead of the rest of the world should be enough to en­able a de­ci­sive strate­gic ad­van­tage.

The ques­tion is, how likely is it that one na­tion/​pro­ject/​AI could get that far ahead of ev­ery­one else? After all, it didn’t hap­pen in the era of the In­dus­trial Revolu­tion. While we did see a mas­sive con­cen­tra­tion of power into a few na­tions on the lead­ing edge of tech­nolog­i­cal ca­pa­bil­ity, there were always at least a few such na­tions and they kept each other in check.

The “surely not faster than the rest of the world com­bined” ar­gu­ment

Some­times I have ex­changes like this:

  • Me: De­ci­sive strate­gic ad­van­tage is plau­si­ble!

  • In­ter­locu­tor: What? That means one en­tity must have more in­no­va­tion power than the rest of the world com­bined, to be able to take over the rest of the world!

  • Me: Yeah, and that’s pos­si­ble af­ter in­tel­li­gence ex­plo­sion. A su­per­in­tel­li­gence would to­tally have that prop­erty.

  • In­ter­locu­tor: Well yeah, if we dropped a su­per­in­tel­li­gence into a world full of hu­mans. But re­al­is­ti­cally the rest of the world will be un­der­go­ing in­tel­li­gence ex­plo­sion too. And in­deed the world as a whole will un­dergo a faster in­tel­li­gence ex­plo­sion than any par­tic­u­lar pro­ject could; to think that one pro­ject could pull ahead of ev­ery­one else is to think that, prior to in­tel­li­gence ex­plo­sion, there would be a sin­gle pro­ject in­no­vat­ing faster than the rest of the world com­bined!

This sec­tion re­sponds to that by way of sketch­ing how one na­tion/​pro­ject/​AI might get 3 − 0.3 years ahead of ev­ery­one else.

Toy model: There are pro­jects which re­search tech­nol­ogy, each with their own “in­no­va­tion rate” at which they pro­duce in­no­va­tions from some la­tent tech tree. When they pro­duce in­no­va­tions, they choose whether to make them pub­lic or pri­vate. They have ac­cess to their pri­vate in­no­va­tions + all the pub­lic in­no­va­tions.

It fol­lows from the above that the pro­ject with ac­cess to the most in­no­va­tions at any given time will be the pro­ject that has the most hoarded in­no­va­tions, even though the set of other pro­jects has a higher com­bined in­no­va­tion rate and also a larger com­bined pool of ac­cessible in­no­va­tions. More­over, the gap be­tween the lead­ing pro­ject and the sec­ond-best pro­ject will in­crease over time, since the lead­ing pro­ject has a slightly higher rate of pro­duc­tion of hoarded in­no­va­tions, but both pro­jects have ac­cess to the same pub­lic innovations

This model leaves out sev­eral im­por­tant things. First, it leaves out the whole “in­tel­li­gence ex­plo­sion” idea: A pro­ject’s in­no­va­tion rate should in­crease as some func­tion of how many in­no­va­tions they have ac­cess to. Ad­ding this in will make the situ­a­tion more ex­treme and make the gap be­tween the lead­ing pro­ject and ev­ery­one else grow even big­ger very quickly.

Se­cond, it leaves out rea­sons why in­no­va­tions might be made pub­lic. Real­is­ti­cally there are three rea­sons: Leaks, spies, and sel­l­ing/​us­ing-in-a-way-that-makes-it-easy-to-copy.

Claim 3: Leaks & Spies: I claim that the 10x-100x speedup Paul prophe­cies will not come with an as­so­ci­ated 10x-100x in­crease in the rate of leaks and suc­cess­ful spy­ing. In­stead the rate of leaks and suc­cess­ful spy­ing will be only a bit higher than it cur­rently is.

This is be­cause hu­mans are still hu­mans even in this soft take­off fu­ture, still in hu­man in­sti­tu­tions like com­pa­nies and gov­ern­ments, still us­ing more or less the same in­ter­net in­fras­truc­ture, etc. New AI-re­lated tech­nolo­gies might make leak­ing and spy­ing eas­ier than it cur­rently is, but they also might make it harder. I’d love to see an in-depth ex­plo­ra­tion of this ques­tion be­cause I don’t feel par­tic­u­larly con­fi­dent.

But any­how, if it doesn’t get much eas­ier than it cur­rently is, then go­ing 3 years to 0.3 years with­out a leak is pos­si­ble, and more gen­er­ally it’s pos­si­ble for the world’s lead­ing pro­ject to build up a 0.3-3 year lead over the sec­ond-place pro­ject. For ex­am­ple, the USSR had spies em­bed­ded in the Man­hat­tan Pro­ject but it still took them 4 more years to make their first bomb.

Claim 4: Sel­ling etc. I claim that the 10x-100x speedup Paul prophe­cies will not come with an as­so­ci­ated 10x-100x in­crease in the bud­get pres­sure on pro­jects to make money fast. Again, to­day AI com­pa­nies reg­u­larly go years with­out turn­ing a profit—Deep­Mind, for ex­am­ple, has never turned a profit and is los­ing some­thing like a billion dol­lars a year for its par­ent com­pany—and I don’t see any par­tic­u­larly good rea­son to ex­pect that to change much.

So yeah, it seems to me that it’s to­tally pos­si­ble for the lead­ing AI pro­ject to sur­vive off in­vestor money and par­ent com­pany money (or gov­ern­ment money, for that mat­ter!) for five years or so, while also keep­ing the rate of leaks and spies low enough that the dis­tance be­tween them and their near­est com­peti­tor in­creases rather than de­creases. (Note how this doesn’t in­volve them “in­no­vat­ing faster than the rest of the world com­bined.”)

Sup­pose they could get a 3-year lead this way, at the peak of their lead. Is that enough?

Well, yes. A 3-year lead dur­ing a time 10x-100x faster than the In­dus­trial Revolu­tion would be like a 30-300 year lead dur­ing the era of the In­dus­trial Revolu­tion. As I ar­gued in the pre­vi­ous sec­tion, even the low end of that range is prob­a­bly enough to get a de­ci­sive strate­gic ad­van­tage.

If this is so, why didn’t na­tions dur­ing the In­dus­trial Revolu­tion try to hoard their in­no­va­tions and gain de­ci­sive strate­gic ad­van­tage?

England ac­tu­ally did, if I re­call cor­rectly. They passed laws and stuff to pre­vent their early In­dus­trial Revolu­tion tech­nol­ogy from spread­ing out­side their bor­ders. They were un­suc­cess­ful—spies and en­trepreneurs dodged the cus­toms offi­cials and snuck blueprints and ex­per­tise out of the coun­try. It’s not sur­pris­ing that they weren’t able to suc­cess­fully hoard in­no­va­tions for 30+ years! En­tire economies are a lot more leaky than AI pro­jects.

What a “Paul Slow” soft take­off might look like ac­cord­ing to me

At some point early in the tran­si­tion to much faster in­no­va­tion rates, the lead­ing AI com­pa­nies “go quiet.” Sev­eral of them ei­ther get huge in­vest­ments or are na­tion­al­ized and given effec­tively un­limited fund­ing. The world as a whole con­tinues to in­no­vate, and the lead­ing com­pa­nies benefit from this pub­lic re­search, but they hoard their own in­no­va­tions to them­selves. Mean­while the benefits of these AI in­no­va­tions are start­ing to be felt; all pro­jects have sig­nifi­cantly in­creased (and con­stantly in­creas­ing) rates of in­no­va­tion. But the fastest in­creases go to the lead­ing pro­ject, which is one year ahead of the sec­ond-best pro­ject. (This sort of gap is nor­mal for tech pro­jects to­day, es­pe­cially the rare mas­sively-funded ones, I think.) Per­haps via a com­bi­na­tion of spy­ing, sel­l­ing, and leaks, that lead nar­rows to six months mid­way through the pro­cess. But by that time things are mov­ing so quickly that a six months’ lead is like a 15-150 year lead dur­ing the era of the In­dus­trial Revolu­tion. It’s not guaran­teed and per­haps still not prob­a­ble, but at least it’s rea­son­ably likely that the lead­ing pro­ject will be able to take over the world if it chooses to.


Ob­jec­tion: What about coal­i­tions? Dur­ing the in­dus­trial rev­olu­tion, if one coun­try did suc­cess­fully avoid all leaks, the other coun­tries could unite against them and make the “pub­lic” tech­nol­ogy in­ac­cessible to them. (Trade does some­thing like this au­to­mat­i­cally, since re­fus­ing to sell your tech­nol­ogy also low­ers your in­come which low­ers your in­no­va­tion rate as a na­tion.)

Re­ply: Coal­i­tions to share AI re­search progress will be harder than free-trade /​ em­bargo coal­i­tions. This is be­cause AI re­search progress is much more the re­sult of rare smart in­di­vi­d­u­als talk­ing face-to-face with each other and much less the re­sult of a zillion differ­ent ac­tions of mil­lions of differ­ent peo­ple, as the econ­omy is. Be­sides, a suc­cess­ful coal­i­tion can be thought of as just an­other pro­ject, and so it’s still true that one pro­ject could get a de­ci­sive strate­gic ad­van­tage. (Is it fair to call “The en­tire world econ­omy” a pro­ject with a de­ci­sive strate­gic ad­van­tage to­day? Well, maybe… but it feels a lot less ac­cu­rate since al­most ev­ery­one is part of the econ­omy but only a few peo­ple would have con­trol of even a broad coal­i­tion AI pro­ject.)


Any­how, those are my thoughts. Not su­per con­fi­dent in all this, but it does feel right to me. Again, the con­clu­sion is not that one pro­ject will take over the world even in Paul’s fu­ture, but rather that such a thing might still hap­pen even in Paul’s fu­ture.



Thanks to Mag­nus Vind­ing for helpful con­ver­sa­tion.