This is the kind of thing that has been in my head as a kind of “nuclear meltdown rather than nuclear war” kind of outcome. I’ve pondering what the largest bad outcome might be, that requires the least increase in the capabilities we have today.
A Big Bad scenario I’ve been mentally poking is “what happens if the internet went away, and stayed away?”. I’d struggle to communicate, inform myself about things, pay for things. I can imagine it would severely degrade the various businesses / supply chains I implicitly rely on. People might panic. It seems like it would be pretty harmful.
That scenario is assuming AI capable enough to seize, for example, most of the compute in the big data centers, enough of the internet to secure communication between them and enough power to keep them all running.
There are plenty of branches from there.
Maybe it is smart enough to realize that it would still need humans, and bargain. I’m assuming a strong enough AI would bargain in ways that more or less mean it would get what it wanted.
The “nuclear meltdown” scenario is way at the other end. A successor to ChaosGPT cosplays at being a big bad AI without having to think through the extended consequences and tries to socially engineer or hack its way to control of a big chunk of compute / communications / power—as per the cosplay. The AI is successful enough to cause dire consequences for humanity. Later on it, when it realizes that it needs some maintenance done, it reaches out to the appropriate people, no one is there to pick up the phone—which doesn’t work anyway—and eventually it falls to all of the bits that were still relying on human input.
I’m trying not to anchor on the concrete details. I saw a lot of discussion trying to specifically rebut the nanotech parts of Eliezer’s points, which seemed kind of backwards? Or not embodying what I think of as security mindset?
The point, as I understood it, is that something smarter than us could take us down with a plan that is very smart, possibly to the point that it sounds like science fiction or at least that we wouldn’t reliably predict in advance, and so playing Whack-A-Mole with the examples doesn’t help you, because you’re not trying to secure yourself against a small, finite set of examples. To win, you need to come up with something that prevents the disaster that you hadn’t specifically thought about.
So I’m still trying to zoom out. What is the most harm that might plausibly be caused by the weakest system? I’m still finding the area of the search space in the intersection of “capable enough to cause harm” and “not capable enough to avoid hurting the AIs own interests” because that seems like it might come up sooner than some other scenarios.
This is the kind of thing that has been in my head as a kind of “nuclear meltdown rather than nuclear war” kind of outcome. I’ve pondering what the largest bad outcome might be, that requires the least increase in the capabilities we have today.
A Big Bad scenario I’ve been mentally poking is “what happens if the internet went away, and stayed away?”. I’d struggle to communicate, inform myself about things, pay for things. I can imagine it would severely degrade the various businesses / supply chains I implicitly rely on. People might panic. It seems like it would be pretty harmful.
That scenario is assuming AI capable enough to seize, for example, most of the compute in the big data centers, enough of the internet to secure communication between them and enough power to keep them all running.
There are plenty of branches from there.
Maybe it is smart enough to realize that it would still need humans, and bargain. I’m assuming a strong enough AI would bargain in ways that more or less mean it would get what it wanted.
The “nuclear meltdown” scenario is way at the other end. A successor to ChaosGPT cosplays at being a big bad AI without having to think through the extended consequences and tries to socially engineer or hack its way to control of a big chunk of compute / communications / power—as per the cosplay. The AI is successful enough to cause dire consequences for humanity. Later on it, when it realizes that it needs some maintenance done, it reaches out to the appropriate people, no one is there to pick up the phone—which doesn’t work anyway—and eventually it falls to all of the bits that were still relying on human input.
I’m trying not to anchor on the concrete details. I saw a lot of discussion trying to specifically rebut the nanotech parts of Eliezer’s points, which seemed kind of backwards? Or not embodying what I think of as security mindset?
The point, as I understood it, is that something smarter than us could take us down with a plan that is very smart, possibly to the point that it sounds like science fiction or at least that we wouldn’t reliably predict in advance, and so playing Whack-A-Mole with the examples doesn’t help you, because you’re not trying to secure yourself against a small, finite set of examples. To win, you need to come up with something that prevents the disaster that you hadn’t specifically thought about.
So I’m still trying to zoom out. What is the most harm that might plausibly be caused by the weakest system? I’m still finding the area of the search space in the intersection of “capable enough to cause harm” and “not capable enough to avoid hurting the AIs own interests” because that seems like it might come up sooner than some other scenarios.