Modest Superintelligences

I’m skep­ti­cal about try­ing to build FAI, but not about try­ing to in­fluence the Sin­gu­lar­ity in a pos­i­tive di­rec­tion. Some peo­ple may be skep­ti­cal even of the lat­ter be­cause they don’t think the pos­si­bil­ity of an in­tel­li­gence ex­plo­sion is a very likely one. I sug­gest that even if in­tel­li­gence ex­plo­sion turns out to be im­pos­si­ble, we can still reach a pos­i­tive Sin­gu­lar­ity by build­ing what I’ll call “mod­est su­per­in­tel­li­gences”, that is, su­per­in­tel­li­gent en­tities, ca­pa­ble of tak­ing over the uni­verse and pre­vent­ing ex­is­ten­tial risks and Malthu­sian out­comes, whose con­struc­tion does not re­quire fast re­cur­sive self-im­prove­ment or other ques­tion­able as­sump­tions about the na­ture of in­tel­li­gence. This helps to es­tab­lish a lower bound on the benefits of an or­ga­ni­za­tion that aims to strate­gi­cally in­fluence the out­come of the Sin­gu­lar­ity.

  • MSI-1: 105 biolog­i­cally cloned hu­mans of von Neu­mann-level in­tel­li­gence, highly ed­u­cated and in­doc­tri­nated from birth to work col­lab­o­ra­tively to­wards some goal, such as build­ing MSI-2 (or equiv­a­lent)

  • MSI-2: 1010 whole brain em­u­la­tions of von Neu­mann, each run­ning at ten times hu­man speed, with WBE-en­abled in­sti­tu­tional con­trols that in­crease group co­her­ence/​ra­tio­nal­ity (or equiv­a­lent)

  • MSI-3: 1020 copies of von Neu­mann WBE, each run­ning at a thou­sand times hu­man speed, with more ad­vanced (to be in­vented) in­sti­tu­tional con­trols and col­lab­o­ra­tion tools (or equiv­a­lent)

(To re­call what the ac­tual von Neu­mann, who we might call MSI-0, ac­com­plished, open his Wikipe­dia page and scroll through the “known for” side­bar.)

Build­ing a MSI-1 seems to re­quire a to­tal cost on the or­der of $100 billion (as­sum­ing $10 mil­lion for each clone), which is com­pa­rable to the Apollo pro­ject, and about 0.25% of the an­nual Gross World Product. (For fur­ther com­par­i­son, note that Ap­ple has a mar­ket cap­i­tal­iza­tion of $561 billion, and an­nual profit of $25 billion.) In ex­change for that cost, any na­tion that un­der­takes the pro­ject has a rea­son­able chance of ob­tain­ing an in­sur­mountable lead in what­ever tech­nolo­gies end up driv­ing the Sin­gu­lar­ity, and with that a large mea­sure of con­trol over its out­come. If no bet­ter strate­gic op­tions come along, lob­by­ing a gov­ern­ment to build MSI-1 and/​or in­fluenc­ing its de­sign and aims seems to be the least that a Sin­gu­lar­i­tar­ian or­ga­ni­za­tion could do.