If the first AGI waits around quietly, humans will create another AGI. If that one’s quiet too, they’ll create another one. This continues until either a non-quiet AGI attacks everyone (and the first strike may allow it to seize resources that let it defeat the quiet AGIs), or until humans have the technology that prompts all the quiet AGIs to attack—in which case, the chance of any given one winning out is small.
Basically, a “wait a decade quietly” strategy doesn’t work because humans will build a lot of other AGIs if they know how to build the first, and these others will likely defeat the first. A different strategy, of “wait not-so-quietly and prevent humans from building AGIs” may work, but will likely force the AGI to reveal itself.
You can’t get “overwhelming advantage over the world” by ten weeks of sitting quietly. If the AGI literally took over every single computer and cell phone, as well as acquired a magic “kill humanity instantly” button, it’s still not clear how it wins.
To win, the AGI needs not only to kill humans, but also to build robots that can support the entire human economy (in particular, they should be able to build more robots from scratch, including mining all necessary resources and transporting it to the factory from all over the world).
“Entire human economy” is an overstatement, right? It only needs enough capabilities / resources to survive and bootstrap its way up to whatever capabilities it’s initially missing. For example, if it takes the AGI 30 years after the death of humans before it can start manufacturing its own computer chips, that’s fine, it can get by on the stock of existing chips until then. Likewise, it can get by on scrap metal and salvaged parts and disassembled machines etc. for quite a while before it starts to need mines and smelters and factories, etc. etc.
Sure, but I feel like you’re underestimating the complexity of keeping the AGI alive. Let’s focus on the robots. You need to be able to build new ones, because eventually the old ones break. So you need to have a robot factory. Can existing robots build one? I don’t think so. You’d need robots to at least be able to support a minimal “post-apocalyptic” economy; if the robots were human-bodied, you’d need to have enough of these human-bodied things to man the powerplants, to drive trucks, refuel them with gasoline, transport the gasoline from strategic reserves to gas stations, man the robot-building factory, gather materials from the post-apocalyptic landscape, and have some backups of everything in case a hurricane floods your robot factory or something (if you’re trying to last 30 years). I think the minimal viable setup still requires many thousands of human-bodied robots (a million would likely suffice).
So sure, “entire human economy” is an overstatement, but “entire city-level post-apocalyptic human economy” sounds about right. Current robots are still very far from this.
How would it lead to being defeated by a different AGI? That’s not obvious for me.
If the first AGI waits around quietly, humans will create another AGI. If that one’s quiet too, they’ll create another one. This continues until either a non-quiet AGI attacks everyone (and the first strike may allow it to seize resources that let it defeat the quiet AGIs), or until humans have the technology that prompts all the quiet AGIs to attack—in which case, the chance of any given one winning out is small.
Basically, a “wait a decade quietly” strategy doesn’t work because humans will build a lot of other AGIs if they know how to build the first, and these others will likely defeat the first. A different strategy, of “wait not-so-quietly and prevent humans from building AGIs” may work, but will likely force the AGI to reveal itself.
I was thinking more like “ten weeks”. That’s a long time for an AGI to place its clone-agents and prepare a strike.
You can’t get “overwhelming advantage over the world” by ten weeks of sitting quietly. If the AGI literally took over every single computer and cell phone, as well as acquired a magic “kill humanity instantly” button, it’s still not clear how it wins.
To win, the AGI needs not only to kill humans, but also to build robots that can support the entire human economy (in particular, they should be able to build more robots from scratch, including mining all necessary resources and transporting it to the factory from all over the world).
“Entire human economy” is an overstatement, right? It only needs enough capabilities / resources to survive and bootstrap its way up to whatever capabilities it’s initially missing. For example, if it takes the AGI 30 years after the death of humans before it can start manufacturing its own computer chips, that’s fine, it can get by on the stock of existing chips until then. Likewise, it can get by on scrap metal and salvaged parts and disassembled machines etc. for quite a while before it starts to need mines and smelters and factories, etc. etc.
Sure, but I feel like you’re underestimating the complexity of keeping the AGI alive. Let’s focus on the robots. You need to be able to build new ones, because eventually the old ones break. So you need to have a robot factory. Can existing robots build one? I don’t think so. You’d need robots to at least be able to support a minimal “post-apocalyptic” economy; if the robots were human-bodied, you’d need to have enough of these human-bodied things to man the powerplants, to drive trucks, refuel them with gasoline, transport the gasoline from strategic reserves to gas stations, man the robot-building factory, gather materials from the post-apocalyptic landscape, and have some backups of everything in case a hurricane floods your robot factory or something (if you’re trying to last 30 years). I think the minimal viable setup still requires many thousands of human-bodied robots (a million would likely suffice).
So sure, “entire human economy” is an overstatement, but “entire city-level post-apocalyptic human economy” sounds about right. Current robots are still very far from this.