I hope that the reasoning in my twoposts shows that the AGI has a chance to end up relying on the entire human-built energy industry just to solve as many problems (and hopefully even less) as the millions of humans who work there. An AI trying to take over the world might: a) end up relying on those who work in the energy industry and having to save them from dying and b) be forced to reject any course of action that requires it to destroy the energy sources, as any nuclear war does.
The other way for AGI to destroy humanity while remaining highly sapient requires the AI to solve even more challenges which include the possibility that gray goo isn’t more efficient than life and the fact that the replicators have to be well managed and supplied to build the computing machines (e.g. mammals require at least days of growing and millions of years of evolution!) The entire set of physicists is within half of an OOM from a million, which might even prevent the AGI from accelerating the progress necessary for the goo to arrive.
I hope that the reasoning in my two posts shows that the AGI has a chance to end up relying on the entire human-built energy industry just to solve as many problems (and hopefully even less) as the millions of humans who work there. An AI trying to take over the world might: a) end up relying on those who work in the energy industry and having to save them from dying and b) be forced to reject any course of action that requires it to destroy the energy sources, as any nuclear war does.
The other way for AGI to destroy humanity while remaining highly sapient requires the AI to solve even more challenges which include the possibility that gray goo isn’t more efficient than life and the fact that the replicators have to be well managed and supplied to build the computing machines (e.g. mammals require at least days of growing and millions of years of evolution!) The entire set of physicists is within half of an OOM from a million, which might even prevent the AGI from accelerating the progress necessary for the goo to arrive.