If I’m, say, building a dam, I do not particularly need to think about the bears which formerly lived in the flooded forest. It’s not like the bears are clever enough to think “ah, it’s the dam that’s the problem, let’s go knock it down”. The bears are forced out and can’t do a damn thing about it, because they do not understand why the forest is flooded.
I wouldn’t be shocked if humans can tell their metaphorical forest is flooding before the end. But I don’t think they’ll understand what’s causing it, or have any idea where to point the nukes, or even have any idea that nukes could solve the problem. I mean, obviously there will be people yelling “It’s the AI! We must shut it down!”, but there will also be people shouting a hundred other things, as well as people shouting that only the AI can save us.
This story was based on a somewhat different prompt (it assumed the AI is trying to kill us and that the AI doesn’t foom to nanotech overnight), but I think the core mood is about right:
Like, one day the AGI is throwing cupcakes at a puppy in a very precisely temperature-controlled room. A few days later, a civil war breaks out in Brazil. Then 2 million people die of an unusually nasty flu, and also it’s mostly the 2 million people who are best at handling emergencies but that won’t be obvious for a while, because of course first responders are exposed more than most. At some point there’s a Buzzfeed article on how, through a series of surprising accidents, a puppy-cupcake meme triggered the civil war in Brazil, but this is kind of tongue-in-cheek and nobody’s taking it seriously and also not paying attention because THE ANTARCTIC ICE CAP JUST MELTED which SURE IS ALARMING but it’s actually just a distraction and the thing everybody should have paid attention to is the sudden shift in the isotope mix of biological nitrogen in algae blooms but that never made the mainstream news at all and page 1 of every news source is all about the former Antarctic ice cap right up until the corn crop starts to fail and the carrying capacity of humanity’s food supply drops by 70% overnight.
Why do you expect that the most straightforward plan for an AGI to accumulate resources is so illegible to humans? If the plan is designed to be hidden to humans, then it involves modeling them and trying to deceive them. But if not, then it seems extremely unlikely to look like this, as opposed to the much simpler plan of building a server farm. To put it another way, if you planned using a world model as if humans didn’t exist, you wouldn’t make plans involving causing a civil war in Brazil. Unless you expect the AI to be modeling the world at an atomic level, which seems computationally intractable particularly for a machine with the computational resources of the first AGI.
This. Any realistic takeoff with classical computers cannot rely on simulating the world atomically for taking over the world thanks to Landauer limit being so bounding. It either has very good models of humans and deceptive capabilities (Which I think are likely), Or it doesn’t win.
You are postulating perpetual motion machines in AI form or you think Quantum Computers are likely to be practical this century.
If I’m, say, building a dam, I do not particularly need to think about the bears which formerly lived in the flooded forest. It’s not like the bears are clever enough to think “ah, it’s the dam that’s the problem, let’s go knock it down”. The bears are forced out and can’t do a damn thing about it, because they do not understand why the forest is flooded.
I wouldn’t be shocked if humans can tell their metaphorical forest is flooding before the end. But I don’t think they’ll understand what’s causing it, or have any idea where to point the nukes, or even have any idea that nukes could solve the problem. I mean, obviously there will be people yelling “It’s the AI! We must shut it down!”, but there will also be people shouting a hundred other things, as well as people shouting that only the AI can save us.
This story was based on a somewhat different prompt (it assumed the AI is trying to kill us and that the AI doesn’t foom to nanotech overnight), but I think the core mood is about right:
Why do you expect that the most straightforward plan for an AGI to accumulate resources is so illegible to humans? If the plan is designed to be hidden to humans, then it involves modeling them and trying to deceive them. But if not, then it seems extremely unlikely to look like this, as opposed to the much simpler plan of building a server farm. To put it another way, if you planned using a world model as if humans didn’t exist, you wouldn’t make plans involving causing a civil war in Brazil. Unless you expect the AI to be modeling the world at an atomic level, which seems computationally intractable particularly for a machine with the computational resources of the first AGI.
This. Any realistic takeoff with classical computers cannot rely on simulating the world atomically for taking over the world thanks to Landauer limit being so bounding. It either has very good models of humans and deceptive capabilities (Which I think are likely), Or it doesn’t win.
You are postulating perpetual motion machines in AI form or you think Quantum Computers are likely to be practical this century.