This seems unlikely to be the case to me. However, even if this is the case and so the AI doesn’t need to deceive us, isn’t disempowering humans via force still necessary? Like, if the AI sets up a server farm somewhere and starts to deploy nanotech factories, we could, if not yet disempowered, literally nuke it. Perhaps this exact strategy would fail for various reasons, but more broadly, if the AI is optimizing for gaining resources/accomplishing its goals as if humans did not exist, then it seems unlikely to be able to defend against human attacks. For example, if we think about the ants analogy, ants are incapable of harming us not just because they are stupid, but because they are also extremely physically weak. If human are faced with physically powerful animals, even if we can subdue them easily, we still have to think about them to do it.
If I’m, say, building a dam, I do not particularly need to think about the bears which formerly lived in the flooded forest. It’s not like the bears are clever enough to think “ah, it’s the dam that’s the problem, let’s go knock it down”. The bears are forced out and can’t do a damn thing about it, because they do not understand why the forest is flooded.
I wouldn’t be shocked if humans can tell their metaphorical forest is flooding before the end. But I don’t think they’ll understand what’s causing it, or have any idea where to point the nukes, or even have any idea that nukes could solve the problem. I mean, obviously there will be people yelling “It’s the AI! We must shut it down!”, but there will also be people shouting a hundred other things, as well as people shouting that only the AI can save us.
This story was based on a somewhat different prompt (it assumed the AI is trying to kill us and that the AI doesn’t foom to nanotech overnight), but I think the core mood is about right:
Like, one day the AGI is throwing cupcakes at a puppy in a very precisely temperature-controlled room. A few days later, a civil war breaks out in Brazil. Then 2 million people die of an unusually nasty flu, and also it’s mostly the 2 million people who are best at handling emergencies but that won’t be obvious for a while, because of course first responders are exposed more than most. At some point there’s a Buzzfeed article on how, through a series of surprising accidents, a puppy-cupcake meme triggered the civil war in Brazil, but this is kind of tongue-in-cheek and nobody’s taking it seriously and also not paying attention because THE ANTARCTIC ICE CAP JUST MELTED which SURE IS ALARMING but it’s actually just a distraction and the thing everybody should have paid attention to is the sudden shift in the isotope mix of biological nitrogen in algae blooms but that never made the mainstream news at all and page 1 of every news source is all about the former Antarctic ice cap right up until the corn crop starts to fail and the carrying capacity of humanity’s food supply drops by 70% overnight.
Why do you expect that the most straightforward plan for an AGI to accumulate resources is so illegible to humans? If the plan is designed to be hidden to humans, then it involves modeling them and trying to deceive them. But if not, then it seems extremely unlikely to look like this, as opposed to the much simpler plan of building a server farm. To put it another way, if you planned using a world model as if humans didn’t exist, you wouldn’t make plans involving causing a civil war in Brazil. Unless you expect the AI to be modeling the world at an atomic level, which seems computationally intractable particularly for a machine with the computational resources of the first AGI.
This. Any realistic takeoff with classical computers cannot rely on simulating the world atomically for taking over the world thanks to Landauer limit being so bounding. It either has very good models of humans and deceptive capabilities (Which I think are likely), Or it doesn’t win.
You are postulating perpetual motion machines in AI form or you think Quantum Computers are likely to be practical this century.
Basically this. A human that fights against powerful animals like gorillas, bears, lions or tigers will mostly lose melee fights without luck or outliers, and even being lucky means you’ve probably gotten seriously injured to the point that you would die if not treated.
If the human thinks about it and brings a gun, the situation is reversed, with animals struggling to defeat humans barring outliers or luck. That’s the power of thinking: Not to enhance your previous skills, but to gain all-new skills.
This seems unlikely to be the case to me. However, even if this is the case and so the AI doesn’t need to deceive us, isn’t disempowering humans via force still necessary? Like, if the AI sets up a server farm somewhere and starts to deploy nanotech factories, we could, if not yet disempowered, literally nuke it. Perhaps this exact strategy would fail for various reasons, but more broadly, if the AI is optimizing for gaining resources/accomplishing its goals as if humans did not exist, then it seems unlikely to be able to defend against human attacks. For example, if we think about the ants analogy, ants are incapable of harming us not just because they are stupid, but because they are also extremely physically weak. If human are faced with physically powerful animals, even if we can subdue them easily, we still have to think about them to do it.
If I’m, say, building a dam, I do not particularly need to think about the bears which formerly lived in the flooded forest. It’s not like the bears are clever enough to think “ah, it’s the dam that’s the problem, let’s go knock it down”. The bears are forced out and can’t do a damn thing about it, because they do not understand why the forest is flooded.
I wouldn’t be shocked if humans can tell their metaphorical forest is flooding before the end. But I don’t think they’ll understand what’s causing it, or have any idea where to point the nukes, or even have any idea that nukes could solve the problem. I mean, obviously there will be people yelling “It’s the AI! We must shut it down!”, but there will also be people shouting a hundred other things, as well as people shouting that only the AI can save us.
This story was based on a somewhat different prompt (it assumed the AI is trying to kill us and that the AI doesn’t foom to nanotech overnight), but I think the core mood is about right:
Why do you expect that the most straightforward plan for an AGI to accumulate resources is so illegible to humans? If the plan is designed to be hidden to humans, then it involves modeling them and trying to deceive them. But if not, then it seems extremely unlikely to look like this, as opposed to the much simpler plan of building a server farm. To put it another way, if you planned using a world model as if humans didn’t exist, you wouldn’t make plans involving causing a civil war in Brazil. Unless you expect the AI to be modeling the world at an atomic level, which seems computationally intractable particularly for a machine with the computational resources of the first AGI.
This. Any realistic takeoff with classical computers cannot rely on simulating the world atomically for taking over the world thanks to Landauer limit being so bounding. It either has very good models of humans and deceptive capabilities (Which I think are likely), Or it doesn’t win.
You are postulating perpetual motion machines in AI form or you think Quantum Computers are likely to be practical this century.
Basically this. A human that fights against powerful animals like gorillas, bears, lions or tigers will mostly lose melee fights without luck or outliers, and even being lucky means you’ve probably gotten seriously injured to the point that you would die if not treated.
If the human thinks about it and brings a gun, the situation is reversed, with animals struggling to defeat humans barring outliers or luck. That’s the power of thinking: Not to enhance your previous skills, but to gain all-new skills.