AGI designs a super-COVID that is more infectious than the latest Omicron variants, persists on surfaces much longer, and is also deadly to >90% of those it infects.
This is outright magical thinking.
It’s more or less exactly of the kind of magical thinking the OP seemed to be complaining about.
The likelihood of that step going wrong is about 99.999 percent.
And doubling down by saying it “should be doable in one day”, as if that were obvious, is ultra-mega-beyond-the-valley-of-magical-thinking.
Being generally intelligent, or even being very intelligent within any realistic bounds, does not imply the ability to do this, let alone to do it quickly. You’re talking about something that could easily beyond the physical limits of perfect use of the computational capacity you’ve granted the AGI. It’s about as credible as confidently asserting that the AGI would invent faster than light travel, create free energy, and solve NP-complete problems in constant time.
In fact, the first thing you could call “AGI” is not going to be even close to making perfect use of either its initial hardware or any hardware it acquires later.
On its initial hardware, it probably won’t be much smarter than a single human, if even that smart. Neither in terms of the complexities of the problems it can solve, nor in terms of the speed with which it can do so. It also won’t be built to scale by running widely distributed with limited internode communcation bandwidth, so it won’t get anything close to linear returns on stolen computing capacity on the Internet. And if it’s built out of ML it will not have an obvious path toward either architectural improvement or intelligence explosion, no matter how much hardware it manages to break into.
And just to be clear, contrary to what you might believe from listening to people on Less Wrong, humans cannot do the kind of offline biological design you suggest, are not close to being able to do it, and do not confidently know any path toward developing the ability. It’s very likely that it can’t be done at all without either iterative real-world testing, or running incredibly massive simulations… more in the nature of “Jupiter brain” massive than of “several tens of percent of the Internet” massive.
AGI uses this hole to take over several tens of percent of all computers attached to the internet.
This would be doable if you already had several tens of percent ofall computers to work with. Figuring out how to do it on the hardware you start with could be nontrivial. And it would be quite noticeable if you didn’t do an extremely good job of working around detection measures of which you would have no prior knowledge. That computing power is in use. Its diversion would be noticed. That matters because even if you could eventually do your step 5, you’d need to compute undisturbed for a long time (much more than a day).
AGI tricks some humans into synthesizing the super-COVID and mailing a sample to the place where its drones are collected.
This could maybe be done in some labs, given a complete “design”. I don’t think you could trick somebody into doing it without them noticing, no matter how smart you were. It would be a very, very obvious thing.
And it is not entirely obvious that there is literally anything you could say to convince anybody to do it once they noticed. Especially not after you’d spooked everybody by doing step 3.
AGI uses the materials left behind in the depopulated cities to make more robots, more drones, more super-COVID, and infect >90% of the survivors.
It’s going to have to make something different, because many of the survivors probably survived because they were congenitally immune to the original. Biology is messy that way. But OK, it could probably clean up the survivors. Or just leave them around and knock them back to the stone age now and then.
Of course, you’re also assuming that the thing is awfully motivated and focused on this particular purpose, to make your plan the very first thing it expends time and assumes risk to do. The probability of that is effectively unknown. Instrumental convergence does not prove it will be so motivated. For one thing, the AGI is very unlikely to be VNM-rational, meaning that it’s not even going to have a coherent utility function. Humans don’t. So all the pretty pseudo-mathematical instrumental convergence arguments are of limited use.
This is outright magical thinking.
It’s more or less exactly of the kind of magical thinking the OP seemed to be complaining about.
The likelihood of that step going wrong is about 99.999 percent.
And doubling down by saying it “should be doable in one day”, as if that were obvious, is ultra-mega-beyond-the-valley-of-magical-thinking.
Being generally intelligent, or even being very intelligent within any realistic bounds, does not imply the ability to do this, let alone to do it quickly. You’re talking about something that could easily beyond the physical limits of perfect use of the computational capacity you’ve granted the AGI. It’s about as credible as confidently asserting that the AGI would invent faster than light travel, create free energy, and solve NP-complete problems in constant time.
In fact, the first thing you could call “AGI” is not going to be even close to making perfect use of either its initial hardware or any hardware it acquires later.
On its initial hardware, it probably won’t be much smarter than a single human, if even that smart. Neither in terms of the complexities of the problems it can solve, nor in terms of the speed with which it can do so. It also won’t be built to scale by running widely distributed with limited internode communcation bandwidth, so it won’t get anything close to linear returns on stolen computing capacity on the Internet. And if it’s built out of ML it will not have an obvious path toward either architectural improvement or intelligence explosion, no matter how much hardware it manages to break into.
And just to be clear, contrary to what you might believe from listening to people on Less Wrong, humans cannot do the kind of offline biological design you suggest, are not close to being able to do it, and do not confidently know any path toward developing the ability. It’s very likely that it can’t be done at all without either iterative real-world testing, or running incredibly massive simulations… more in the nature of “Jupiter brain” massive than of “several tens of percent of the Internet” massive.
This would be doable if you already had several tens of percent ofall computers to work with. Figuring out how to do it on the hardware you start with could be nontrivial. And it would be quite noticeable if you didn’t do an extremely good job of working around detection measures of which you would have no prior knowledge. That computing power is in use. Its diversion would be noticed. That matters because even if you could eventually do your step 5, you’d need to compute undisturbed for a long time (much more than a day).
This could maybe be done in some labs, given a complete “design”. I don’t think you could trick somebody into doing it without them noticing, no matter how smart you were. It would be a very, very obvious thing.
And it is not entirely obvious that there is literally anything you could say to convince anybody to do it once they noticed. Especially not after you’d spooked everybody by doing step 3.
It’s going to have to make something different, because many of the survivors probably survived because they were congenitally immune to the original. Biology is messy that way. But OK, it could probably clean up the survivors. Or just leave them around and knock them back to the stone age now and then.
Of course, you’re also assuming that the thing is awfully motivated and focused on this particular purpose, to make your plan the very first thing it expends time and assumes risk to do. The probability of that is effectively unknown. Instrumental convergence does not prove it will be so motivated. For one thing, the AGI is very unlikely to be VNM-rational, meaning that it’s not even going to have a coherent utility function. Humans don’t. So all the pretty pseudo-mathematical instrumental convergence arguments are of limited use.