I agree there’s lots of uncertainty about what the industrial explosion will look like; AI 2027 was my extremely unconfident best guess. I think this cuts both ways though; I think it’s entirely plausible (>5% likely, though less than 50%) that the army of superintelligences in the datacenter could follow the following strategy in less than three months:
(1) Keep doing the intelligence explosion to create vastly superhuman AIs that are basically qualitatively different from human scientists at the task of figuring out how to rapidly design and build nanobots etc. (2) Build nanobots etc. by doing tons of parallel experiments in various biolabs and physics labs and whatnot around the world (maybe thousands of such facilities could be acquired simply by spending OpenBrain’s money?) The theory will have been worked out beforehand by the godlike AI scientists from step 1, and the experiments won’t be fucking around, they’ll be doing the minimum necessary experiments to efficiently cut down the search space and estimate the relevant parameters to fix the most viable designs, and then building those designs.
Sure sounds wild, sounds sci-fi even. But historically things that sound at least this wild and sci-fi sure do seem to happen sometimes e.g. nukes would have sounded this way to people beforehand, and the industrial revolution (“machines that move themselves and make other machines? Enabling peasants to live like kings after only a half-dozen generations of growth?”) probably would have sounded this way to people in 1500 too. e.g. the modern cell phone would have sounded like this to someone in 1925 or even maybe 1975. And importantly, all these past examples of wild things happening happened with mere humans doing the R&D. I think we should firmly reserve >5% credence for “ASIs will basically just be gods as far as we can tell, just like how humans are basically gods from the perspective of dolphins or monkeys.”
Re: Cooperation from pre-existing companies: I don’t think this is going to really change the basic picture. Taking your example with OpenBrain and ASML. OpenBrain has ASI, no one else does. ASML’s internal data is useful for the ASIs, maybe, and ASML’s machinery is useful to the ASIs, probably. But the ASIs can probably find a workaround if ASML is obstinate. For example they can hack ASML and steal the data. Or they can do a hostile takeover maybe, and buy ASML. As for the machines, I bet they could offer giant gobs of money to ASML’s suppliers to supply OpenBrain instead and then OpenBrain could build their own machines in less than six months. ASML will also be a fallible human institution, vulnerable to OpenBrain’s ASI-powered trickery, persuasion, lobbying, and shenanigans (e.g. bribing some of the employees to quit and join OpenBrain, possibly even bribing the BoD)
And ASML is probably one of the best examples you’ve find, of the closest thing to a monopoly against the power of the AI company. Other stuff (robots, physical manufacturing, raw materials) is already less monopolistic and thus really hard to coordinate against OpenBrain.
Re: money:
But you suggest these “mere $” might not be very useful, if ASI has colonised space and doesn’t care. But $ can be used to buy AI and buy crazy new physical tech. So if other actors have lots of $, they can convert that to ability to colonise the stars themselves.
No, the best AIs and physical tech will be kept by OpenBrain in house, and the second-best will be sold by OpenBrain, and the third-best will be way worse (since by hypothesis OpenBrain has ASI and no one else does). You can’t use your $ to buy AI and crazy new physical tech unless OpenBrain lets you, and they won’t let you if they suspect you’ll be able to use it to colonize the stars yourselves, since OpenBrain wants those stars.
Re: Q about Israel: I was imagining close cooperation between the govt and the company, yes. but also, I think it’s not necessary, I think the company itself could outgrow the rest of the world combined potentially. (Through trading with it. The company produces (a) cash cow products and (b) new sci-fi industrial tech that bootstraps towards a self-sustaining robot economy with a short doubling time, and it sells (a) to get cash to buy raw materials and physical labor with which to produce (b). After it’s produced enough (b), it outgrows the rest of the world combined & all the cash becomes basically worthless.)
Agree ASML is one of the trickiest cases for OpenBrain. Though I imagine there are many parts of the semiconductor supply chain with TSMC-level monopolies (which is less than ASML). And i don’t think hacking will work. These companies already protect themselves from this, knowing that it’s a threat today. Data stored on local physical machines that aren’t internet connected, and in ppl’s heads.
And i think you’ll take many months of delay if you go to ASML’s suppliers rather than ASML. ASLM built their factories using many years worth of output from suppliers. Even with massive efficiency gains over ASML (despite not knowing their trade secrets) it will take you months to replicate.
I agree that the more OpenBrain have super-strategy and super-persuasion stuff, the more likely they can capture all the gains from trade. (And military pressure can help here too, like with colonialism.)
Also, if OpenBrain can prevent anyone else developing ASI for years, e.g. by sabotage, i think they have a massive advantage. Then ASML loses their option of just waiting a few months and trading with someone else. I think this is your strongest argument tbh.
Biggest cruxes imo:
Size of SIE
Speed of industrial explosion when the SIE is finishing
Depends on size of SIE and on how hard it is to build fast-replicating robots
How long OpenBrain has a monopoly on ASI (maybe indefinitely via sabotage)
Depends on size of SIE, gap with laggard, whether they dare to sabotage
Whether OpenBrain can take ~all gains from trade with other complementary companies
Depends on AI persuasion/strategy after the SIE, whether OpenBrain can quickly rediscover all their insights with way fewer experiments, and whether OpenBrain has a sustained monopoly on ASI
I’m curious how confident you are a company with a 6 month lead could outgrow the rest of the world by themselves?
I agree there’s lots of uncertainty about what the industrial explosion will look like; AI 2027 was my extremely unconfident best guess. I think this cuts both ways though; I think it’s entirely plausible (>5% likely, though less than 50%) that the army of superintelligences in the datacenter could follow the following strategy in less than three months:
(1) Keep doing the intelligence explosion to create vastly superhuman AIs that are basically qualitatively different from human scientists at the task of figuring out how to rapidly design and build nanobots etc.
(2) Build nanobots etc. by doing tons of parallel experiments in various biolabs and physics labs and whatnot around the world (maybe thousands of such facilities could be acquired simply by spending OpenBrain’s money?) The theory will have been worked out beforehand by the godlike AI scientists from step 1, and the experiments won’t be fucking around, they’ll be doing the minimum necessary experiments to efficiently cut down the search space and estimate the relevant parameters to fix the most viable designs, and then building those designs.
Sure sounds wild, sounds sci-fi even. But historically things that sound at least this wild and sci-fi sure do seem to happen sometimes e.g. nukes would have sounded this way to people beforehand, and the industrial revolution (“machines that move themselves and make other machines? Enabling peasants to live like kings after only a half-dozen generations of growth?”) probably would have sounded this way to people in 1500 too. e.g. the modern cell phone would have sounded like this to someone in 1925 or even maybe 1975. And importantly, all these past examples of wild things happening happened with mere humans doing the R&D. I think we should firmly reserve >5% credence for “ASIs will basically just be gods as far as we can tell, just like how humans are basically gods from the perspective of dolphins or monkeys.”
Re: Cooperation from pre-existing companies: I don’t think this is going to really change the basic picture. Taking your example with OpenBrain and ASML. OpenBrain has ASI, no one else does. ASML’s internal data is useful for the ASIs, maybe, and ASML’s machinery is useful to the ASIs, probably. But the ASIs can probably find a workaround if ASML is obstinate. For example they can hack ASML and steal the data. Or they can do a hostile takeover maybe, and buy ASML. As for the machines, I bet they could offer giant gobs of money to ASML’s suppliers to supply OpenBrain instead and then OpenBrain could build their own machines in less than six months. ASML will also be a fallible human institution, vulnerable to OpenBrain’s ASI-powered trickery, persuasion, lobbying, and shenanigans (e.g. bribing some of the employees to quit and join OpenBrain, possibly even bribing the BoD)
And ASML is probably one of the best examples you’ve find, of the closest thing to a monopoly against the power of the AI company. Other stuff (robots, physical manufacturing, raw materials) is already less monopolistic and thus really hard to coordinate against OpenBrain.
Re: money:
No, the best AIs and physical tech will be kept by OpenBrain in house, and the second-best will be sold by OpenBrain, and the third-best will be way worse (since by hypothesis OpenBrain has ASI and no one else does). You can’t use your $ to buy AI and crazy new physical tech unless OpenBrain lets you, and they won’t let you if they suspect you’ll be able to use it to colonize the stars yourselves, since OpenBrain wants those stars.
Re: Q about Israel: I was imagining close cooperation between the govt and the company, yes. but also, I think it’s not necessary, I think the company itself could outgrow the rest of the world combined potentially. (Through trading with it. The company produces (a) cash cow products and (b) new sci-fi industrial tech that bootstraps towards a self-sustaining robot economy with a short doubling time, and it sells (a) to get cash to buy raw materials and physical labor with which to produce (b). After it’s produced enough (b), it outgrows the rest of the world combined & all the cash becomes basically worthless.)
Thanks for this!
Yep i agree that scenario gets >5%!
Agree ASML is one of the trickiest cases for OpenBrain. Though I imagine there are many parts of the semiconductor supply chain with TSMC-level monopolies (which is less than ASML). And i don’t think hacking will work. These companies already protect themselves from this, knowing that it’s a threat today. Data stored on local physical machines that aren’t internet connected, and in ppl’s heads.
And i think you’ll take many months of delay if you go to ASML’s suppliers rather than ASML. ASLM built their factories using many years worth of output from suppliers. Even with massive efficiency gains over ASML (despite not knowing their trade secrets) it will take you months to replicate.
I agree that the more OpenBrain have super-strategy and super-persuasion stuff, the more likely they can capture all the gains from trade. (And military pressure can help here too, like with colonialism.)
Also, if OpenBrain can prevent anyone else developing ASI for years, e.g. by sabotage, i think they have a massive advantage. Then ASML loses their option of just waiting a few months and trading with someone else. I think this is your strongest argument tbh.
Biggest cruxes imo:
Size of SIE
Speed of industrial explosion when the SIE is finishing
Depends on size of SIE and on how hard it is to build fast-replicating robots
How long OpenBrain has a monopoly on ASI (maybe indefinitely via sabotage)
Depends on size of SIE, gap with laggard, whether they dare to sabotage
Whether OpenBrain can take ~all gains from trade with other complementary companies
Depends on AI persuasion/strategy after the SIE, whether OpenBrain can quickly rediscover all their insights with way fewer experiments, and whether OpenBrain has a sustained monopoly on ASI
I’m curious how confident you are a company with a 6 month lead could outgrow the rest of the world by themselves?