Why no total winner?

Why doesn’t a sin­gle power rule the world to­day?

[I’m tak­ing ad­van­tage of the new “LW posts as blog posts” for­mat to post some­thing I’m pretty un­sure about. I’m work­ing from my mem­o­ries of the blog posts, and from a dis­cus­sion I had with Robin Han­son and Katja Grace in late 2012. Please let me know if any of this is in­ac­cu­rate!]

One of the key differ­ences of opinion in the Han­son-Yud­kowsky AI-Foom De­bate is about the idea of a “de­ci­sive ad­van­tage”. If I’m not mis­rep­re­sent­ing the par­ties hor­ribly, the idea is that some point in a world with AGI, some AGI-en­abled party uses their greater in­tel­li­gence to in­crease their gen­eral power: money, re­sources, con­trol over oth­ers, in­tel­li­gence and such­like. That greater power in­creases their abil­ity to gain power, re­sult­ing in a snow­ball effect that ends with some party hav­ing con­trol over the out­comes for all of Earth-origi­nat­ing life.

Robin asks the very rea­son­able ques­tion: if that’s how things work, why hasn’t it already hap­pened? What stops the largest busi­ness us­ing its de­ci­sive power over smaller ones to defeat and ab­sorb them, grow­ing ever larger and more pow­er­ful un­til all other busi­nesses fall to its power? Why do we have mul­ti­ple na­tions to­day, when this model would seem to pre­dict that a sin­gle state should ul­ti­mately con­quer and rule all? I don’t re­mem­ber Robin propos­ing an an­swer of his own: a mechanism or the­o­ret­i­cal model that would lead us to ex­pect mul­ti­ple pow­ers. But it seems like a good ques­tion, and it’s bugged me ever since.

I think I’d need to be much more of a stu­dent of his­tory than I am to have any con­fi­dence in an an­swer, so let me share some wild spec­u­la­tion that might at least start dis­cus­sion:

  • Reg­u­la­tion: Such growth isn’t an op­tion for le­gal busi­nesses at all, be­cause states ex­ist. So pow­er­ful a busi­ness would challenge the power of the state, and the state is in a po­si­tion to dis­al­low that. The ex­plicit pur­pose of monopoly leg­is­la­tion is to stop a busi­ness which has be­come very pow­er­ful in one area from lev­er­ag­ing that to be­come pow­er­ful el­se­where.

  • Prin­ci­pal-agent prob­lems: it sure would be eas­ier to keep an em­pire to­gether if you could re­li­ably ap­point gen­er­als and rulers who always did what you told them to. Espe­cially if the round-trip time for get­ting them a mes­sage is on the or­der of weeks, and you have to en­trust them with the dis­cre­tion to wield tremen­dous power in the mean time.

  • Mo­ral norms: Nu­clear weapons gave the USA a de­ci­sive ad­van­tage at the end of WWII. If the USA had been en­tirely ruth­less and bent on power at any cost, it would im­me­di­ately have used that ad­van­tage to crip­ple all ri­vals for world su­per­power and de­clared its ruler­ship of the world.

I don’t ex­pect any of these fac­tors to limit the growth of an AGI. Is there some more gen­eral limit to power beget­ting power that would also af­fect AGI?