This is the kind of thing that has been in my head as a kind of “nuclear meltdown rather than nuclear war” kind of outcome. I’ve pondering what the largest bad outcome might be, that requires the least increase in the capabilities we have today.
A Big Bad scenario I’ve been mentally poking is “what happens if the internet went away, and stayed away?”. I’d struggle to communicate, inform myself about things, pay for things. I can imagine it would severely degrade the various businesses / supply chains I implicitly rely on. People might panic. It seems like it would be pretty harmful.
That scenario is assuming AI capable enough to seize, for example, most of the compute in the big data centers, enough of the internet to secure communication between them and enough power to keep them all running.
There are plenty of branches from there.
Maybe it is smart enough to realize that it would still need humans, and bargain. I’m assuming a strong enough AI would bargain in ways that more or less mean it would get what it wanted.
The “nuclear meltdown” scenario is way at the other end. A successor to ChaosGPT cosplays at being a big bad AI without having to think through the extended consequences and tries to socially engineer or hack its way to control of a big chunk of compute / communications / power—as per the cosplay. The AI is successful enough to cause dire consequences for humanity. Later on it, when it realizes that it needs some maintenance done, it reaches out to the appropriate people, no one is there to pick up the phone—which doesn’t work anyway—and eventually it falls to all of the bits that were still relying on human input.
I’m trying not to anchor on the concrete details. I saw a lot of discussion trying to specifically rebut the nanotech parts of Eliezer’s points, which seemed kind of backwards? Or not embodying what I think of as security mindset?
The point, as I understood it, is that something smarter than us could take us down with a plan that is very smart, possibly to the point that it sounds like science fiction or at least that we wouldn’t reliably predict in advance, and so playing Whack-A-Mole with the examples doesn’t help you, because you’re not trying to secure yourself against a small, finite set of examples. To win, you need to come up with something that prevents the disaster that you hadn’t specifically thought about.
So I’m still trying to zoom out. What is the most harm that might plausibly be caused by the weakest system? I’m still finding the area of the search space in the intersection of “capable enough to cause harm” and “not capable enough to avoid hurting the AIs own interests” because that seems like it might come up sooner than some other scenarios.
I might have missed something, but it looks to me like the first ordering might be phrased like the self improvement and the risk aversion are actually happening simultaneously.
If an AI had the ability to self improve for a couple of years before it developed risk aversion, for instance, I think we end up in the “maximal self improvement” / ’high risk” outcomes.
This seems like a big assumption to me:
If an AI has enough resources and is doing the YOLO version of self-improvement, it doesn’t seem like it necessarily requires much in the way of self-awareness or risk apprehension—particularly if it is willing to burn resources on the task. If you ask a current LLM how to take over the world, it says things that appear like “evil AI cosplay”—I could imagine something like that leading to YOLO self-improvement that has some small risk of stumbling across a gain that starts to compound.
There seem to be a lot of big assumptions in this piece, doing a lot of heavy lifting. Maybe I’ve gotten more used to LW style conversational norms about tagging things as assumptions, and it actually fine? My gut instinct is something like “all of these assumptions stack up to target this to a really thin slice of reality, and I shouldn’t update much on it directly”.