In an Open Letter to Scott Alexander, Stephen Pinker claims RAND [1] found there is “no describable scenario in which an AI could conclusively pose an extinction threat to humanity.”
This is at best sloppy research, and at worst actively misleading, as Cody Fenwick points out that Rand’s conclusions significantly differed from what Pinker claimed.
RAND actually said AI plausibly could wipe out humanity in 2 of 3 scenarios they looked at, though they couldn’t definitively assert that any of the three scenarios is likely or unlikely.
I can see where Pinker got his claim—Rand did not show there is obviously a describable scenario in which an AI wipes out humanity. But:
1) Rand didn’t search very hard
2) RAND certainly doesn’t find it implausible that AI wipes out humanity, which is what Pinker is trying to argue against.
Overall, I continue to be disappointed by Pinker’s arguments.
I think it’s sort of correct to characterize the RAND paper as concluding that extinction risk is not likely, though they add some nice caveats.
But the bigger problem is that the RAND paper is very bad, with a total lack of imagination about what a powerful AI could do. (It doesn’t even consider robots or drones.)
I think it’s weasly to characterise it that way, as the RAND report also goes out of its way to say that they don’t conclude extinction risk is unlikely. Plus, as you say, it’s not a good paper.
In what way did RAND not search very hard? The authors were transparent about their research process and screened specifically for scenarios where it could be plausible that every last human would die. They consulted multiple experts on each topic (except for the malicious geoengineering case, which is a very niche subject for which there are only three published papers on intentional warming). They left out mirror life for another analysis, but the cyber-physical argument that they make for extinction methodology is valid.
AFAICT, they don’t even talk about plain old robots killing everyone. That’s a pretty obvious way to exterminate an enemy, even if it does not look respectable.
For another, they seem to limit themselves to scenarios that only use existing tech in existing ways. Like, why assume the AI can’t ramp up nuclear stockpiles massively? Or, more pertinently, dismiss nanotech, not on the grounds of it taking too long to create, but technical feasibility? AFAICT drexlerian nanotech is pretty feasible, let alone programmable biology.
It has the smell of methodological rigour at the top—but at crucial points of concentration for the argument they make weird choices.
Consider bio risk—which is the one they couldn’t rule out. The reason they didn’t rule it in is primarily because they couldn’t find a plausible AI-initiated deployment mechanism with the current level of technology, yet they put this caveat in the deployment section:
“Without the means to interact in the physical world in a general way (e.g., using robotics or manipulating expert humans to perform certain tasks), it is unclear how an AI agent would carry out these steps.”
Which is disingenuous at best, given proven propensity and efficacy of AI’s to perform persuasion and manipulation. This is made worse by the fact that they consider exactly those vectors in the nuclear scenarios and find them plausible (but the nuclear scenarios suffer from being catastrophic and not extinction at the results level).
In an Open Letter to Scott Alexander, Stephen Pinker claims RAND [1] found there is “no describable scenario in which an AI could conclusively pose an extinction threat to humanity.”
This is at best sloppy research, and at worst actively misleading, as Cody Fenwick points out that Rand’s conclusions significantly differed from what Pinker claimed.
RAND actually said AI plausibly could wipe out humanity in 2 of 3 scenarios they looked at, though they couldn’t definitively assert that any of the three scenarios is likely or unlikely.
I can see where Pinker got his claim—Rand did not show there is obviously a describable scenario in which an AI wipes out humanity. But:
1) Rand didn’t search very hard
2) RAND certainly doesn’t find it implausible that AI wipes out humanity, which is what Pinker is trying to argue against.
Overall, I continue to be disappointed by Pinker’s arguments.
[1] https://www.rand.org/content/dam/rand/pubs/research_reports/RRA3000/RRA3034-1/RAND_RRA3034-1.pdf
I think it’s sort of correct to characterize the RAND paper as concluding that extinction risk is not likely, though they add some nice caveats.
But the bigger problem is that the RAND paper is very bad, with a total lack of imagination about what a powerful AI could do. (It doesn’t even consider robots or drones.)
I think it’s weasly to characterise it that way, as the RAND report also goes out of its way to say that they don’t conclude extinction risk is unlikely. Plus, as you say, it’s not a good paper.
In what way did RAND not search very hard? The authors were transparent about their research process and screened specifically for scenarios where it could be plausible that every last human would die. They consulted multiple experts on each topic (except for the malicious geoengineering case, which is a very niche subject for which there are only three published papers on intentional warming). They left out mirror life for another analysis, but the cyber-physical argument that they make for extinction methodology is valid.
AFAICT, they don’t even talk about plain old robots killing everyone. That’s a pretty obvious way to exterminate an enemy, even if it does not look respectable.
For another, they seem to limit themselves to scenarios that only use existing tech in existing ways. Like, why assume the AI can’t ramp up nuclear stockpiles massively? Or, more pertinently, dismiss nanotech, not on the grounds of it taking too long to create, but technical feasibility? AFAICT drexlerian nanotech is pretty feasible, let alone programmable biology.
It has the smell of methodological rigour at the top—but at crucial points of concentration for the argument they make weird choices.
Consider bio risk—which is the one they couldn’t rule out. The reason they didn’t rule it in is primarily because they couldn’t find a plausible AI-initiated deployment mechanism with the current level of technology, yet they put this caveat in the deployment section:
“Without the means to interact in the physical world in a general way (e.g., using robotics or manipulating expert humans to perform certain tasks), it is unclear how an AI agent would carry out these steps.”
Which is disingenuous at best, given proven propensity and efficacy of AI’s to perform persuasion and manipulation. This is made worse by the fact that they consider exactly those vectors in the nuclear scenarios and find them plausible (but the nuclear scenarios suffer from being catastrophic and not extinction at the results level).