Why do you expect that the most straightforward plan for an AGI to accumulate resources is so illegible to humans? If the plan is designed to be hidden to humans, then it involves modeling them and trying to deceive them. But if not, then it seems extremely unlikely to look like this, as opposed to the much simpler plan of building a server farm. To put it another way, if you planned using a world model as if humans didn’t exist, you wouldn’t make plans involving causing a civil war in Brazil. Unless you expect the AI to be modeling the world at an atomic level, which seems computationally intractable particularly for a machine with the computational resources of the first AGI.
This. Any realistic takeoff with classical computers cannot rely on simulating the world atomically for taking over the world thanks to Landauer limit being so bounding. It either has very good models of humans and deceptive capabilities (Which I think are likely), Or it doesn’t win.
You are postulating perpetual motion machines in AI form or you think Quantum Computers are likely to be practical this century.
Why do you expect that the most straightforward plan for an AGI to accumulate resources is so illegible to humans? If the plan is designed to be hidden to humans, then it involves modeling them and trying to deceive them. But if not, then it seems extremely unlikely to look like this, as opposed to the much simpler plan of building a server farm. To put it another way, if you planned using a world model as if humans didn’t exist, you wouldn’t make plans involving causing a civil war in Brazil. Unless you expect the AI to be modeling the world at an atomic level, which seems computationally intractable particularly for a machine with the computational resources of the first AGI.
This. Any realistic takeoff with classical computers cannot rely on simulating the world atomically for taking over the world thanks to Landauer limit being so bounding. It either has very good models of humans and deceptive capabilities (Which I think are likely), Or it doesn’t win.
You are postulating perpetual motion machines in AI form or you think Quantum Computers are likely to be practical this century.