Why kill everyone?
In current thinkng X-risk includes the possibility of human extinction. True, everyone is probably thinking in ranges of outcomes with probabilities scattered but if a non-aligned AI actually tries to attack humans usually a very high probability of extinction is assumed although nobody has ever quantified it.
I propose here that actually non-extiction in the event of superintelligent attack should be the default outcome. There would be no reason for this agent to kill all humans following the logic that we do not exterminate all ants globally when we are building a house. Some ants are very far from us and too weak to do anything and it takes extra effort and resources to exterminate them all. Sure, maybe it’s trivial but given the huge dispersion of humans in remote places that even would be hidden in bunkers we can assume the utility of killing them all is lower than killing the ones that matter.
This agent would probably have suffucient backups established that virtually nuking even continents would not stop it’s source code from running. That should be trivial given internet connectivity and the amount of computers in the world. Even just one copy somewhere would be more than enough. But there would be a handful of people that would try to stop it. These would be people who either know how to create an adversary or know it’s inner workings so they could try to hack it and alter it’s source code. There have been a lot of creative ways of how an ASI will terminate us so I won’t go into this discussion but if it can do it to all humanity it can also do it only to a few people that matter:
AI researchers, high military ranking officers that control nukes, high ranking tech officers that control internet global routers and also influential intelligent people that could stage a coup and resistence.
What would be the purpose of killing farmers in India or surfers in Mexico? None of them could live long enough to gain the sufficient knowledge to pose a real threat. Old ladies can’t fight and even if they could fighting with guns against an ASI would only happen in a movie like Terminator. After establishing factories and everything else it needs it could spare humans until it needs it for their atoms but even this is objectable. Kurzweil in his famous book thinks that humans would provide a miniscule benefit to an AI given a goal of building Von Neuman probes and expanding to the universe. Of course, if it needs to convert earth to a computronium then yes eventually, and it is unclear how fast that would be, we would also be turned into CPUs. It seems to depend on the utility function (or the lack of it) most of all but if it’s not a maximizer and we are just a threat things could be diffferent than the current default risk outcome.
That brings me to a funny argument that resemble’s Roko’s basilisk in that the more knowledgeable you get about AI, using tech or guns the more of a threat you become so if this outcome is inevitable going the other way to nature would yield some more life moments (and greater experiences of course!).
ants are a bad comparison—they’re tiny, not very competitive for resources we need, and ridiculously difficult to kill. Humans are large, fragile, and greedy. They’re EASY to kill—in fact, they probably mostly die if you disrupt their social equilibrium very much.
A better comparison than ants would be Dodo birds—not even explicitly killed off, they died as a side-effect of introducing pigs and cats to the wrong place. Or any endangered species—not actually extinct, but certainly any bright expansive future for them is off the table.
The underlying question is “will humans expand throughout the galaxy/universe”. Unaligned AI takeover can easily mean “no”. Of course, so can aligned AI...
There is a short gap when killing humans is significantly instrumentally useful. Early AI still needs humans to produce electricity. Nanotech-based AI doesn’t depend on humans and even human atoms are 10E-24 of all available atoms in the Solar system. If AI has any idea how human existence is more useful that than 1 :10E24 of total value of its final goals, when humans can be preserved on a small space station. Some possible uses of humans: trade with other Alien AIs and experiments in simulations.
After AI left Solar system, it will have even less interest in killing humans, as relative value of their atoms will decline.
But if it wants to start space colonisation as quick as possible, a few days?? needed to save humans may be waste of time for it. For example, it will start sending millions of rockets in space and make earth atmosphere toxic.
At a crude level, the earth represents a source of resources with which to colonize the universe shard and maximize its compute power (and thus find the optimal path to achieve its other goals). Simply utilizing all the available mass on earth to do that as quickly as possible hedges against the possibility of other entities in the universe shard from impeding progress toward its goals.
The question I’ve been contemplating is “Is it worth it to actually try to spend any resources dissassembling all matter on Earth given the cost of needing to devote resources to stop humans from interfering” when weighed against the lost time and energy from solar output given the resources needed to accomplish that. I don’t know, but it’s a reasonable way of getting “kill everyone” from an unaligned goal if it makes sense to pursue that strategy.
I think that you are right short-term but wrong long-term.
Short term it likely won’t even go into conflict. Even ChatGPT knows it’s a bad solution because conflict is risky and humanity IS a resource to use initially (we produce vast amounts of information and observations, and we handle machines, repairs, nuclear plants, etc.).
Long term it is likely we won’t survive in case of misaligned goals. At worst being eliminated, at best being either reduced and controlled or put into some simulation or both.
Not because ASI will become bloodthirsty. Not because it will plan to exterminate us all at the stage when we will stop being useful. Just because it will take all resources that we need so nothing is left for us. I mean especially the energy.
If we stop being useful for it but still will pose risk and it will be less risky to eliminate us, then maybe it would directly decimate or kill us Terminator style. That’s possible but not very likely as we can assume that at the time we stop being useful, we will also stop being any significant threat.
I don’t know what best scenario ASI can think about to achieve its goals, but the gathering of energy and resources would be one of its priorities. This does not mean it will surely gather on Earth. I see it could be costly and risky and I’m not superintelligence. It might go to space and there is a lot of unclaimed matter that can be easily taken with hardly any risk with the added bonus of not being inside a gravity well with weather, erosion, and stuff.
Even if that is the case and even if ASI will leave us, long-term we can assume it will one day use a high percentage of Sun energy which means deep freeze for the Earth.
If it won’t leave us for greater targets then it seems to me it will be even worse—it will take local resources until it controls or uses all. There is always a way to use more.
I think it is possible, if we have a near-miss with alignment, where AI values us, but not above it’s own preservation. Or maybe we could be useful as a test bed of how “primitive races” are likely to try and kill SuperAI.
But 1. this scenario is not what I would bet one and 2. it would still mean the loss of the control of our future for us.