My point is I don’t think they’re incorrect. All those things are ALSO problems, and many are paths to X-risk even, which I’d consider more likely (in a slow takeoff scenario) than FOOM. A few possible scenarios:
designer pathogens are the obvious example because there’s so many ways they can cause targeted human extinction without risking the AI’s integrity in any way, so they’re obvious candidates for both a misuse of AI by malicious humans and for a rogue AI that is actively trying to kill us for whatever reason. Plus there’s also the related risk of designer microorganisms that alter the Earth’s biosphere as a whole, which would be even deadlier (think something like the cyanobacteria that caused the Great Oxidation Event, except now it’d be something that grows out of control and makes the atmosphere deadly)
autonomous weapons are obviously dangerous in the hands of an AI because that’s really handing it our defenses on a silver platter and makes it turning against us much more likely to succeed (and thus a potentially attractive strategy to seek power)
deepfakes or actual even more sophisticated forms of deception could be exploited by an AI to manipulate humans towards carrying out actions that benefit it or allow it to escape confinement and so on. Being able to see through deepfakes and defend against them would be key to the security of important strategic resources
we don’t know what happens socially and economically if AGI really takes everyone’s job. We go into unexplored territory, and more so, we go there from a place where the AGI will probably be owned by a few to begin with. We might hope for glorious post-scarcity abundance but that’s not the only road nor, I fear, the most likely. If the transition goes badly it can weaken our species enough that all AGI needs to do to get rid of us for good is give a little push.
All these things of course are only relevant with relatively weak-ish AGI. If it is possible to have a self-improving AGI that FOOMs in a matter of hours and kills everyone with nanobots, then we’re screwed either way. But we don’t know that’s possible, the slower scenario are also possible and we can at least defend more against them. Other strategies, like “find ways to align the AGI” or “just don’t build the damn thing” prevent all the scenarios anyway. So I don’t think this statement is particularly missing the mark. Remember that the narrower the statement, the less people can endorse it. This is the Minimum Common Denominator: the things that all the signatories, even with probably very different individual viewpoints, can agree for sure are an issue. Which they are. There’s probably more issues, but consensus on those is not as solid.
Misconstruing an incorrect statement with a correct steelman is incorrect. If I say “I’ve discovered a truly marvelous proof that 2+2=3000 that this margin is too small to contain,” and you reply, “Ah, so you are saying 2+2=4, quite right,” then the fact of your inexplicable discussion of a different and correct statement doesn’t make your interpretation of my original incorrect statement correct.
“Extinction from AI” really doesn’t refer to deepfakes, AI leaving us nothing to do, and algorithmic bias. It doesn’t include any of these categories. There is nothing correct about interpreting “extinction from AI” as referring to either of those things. This holds even if extinction from AI is absolutely impossible, and those other things are both real/imminent and extremely important. Words have meaning even when they refer to fictional ideas.
Deepfakes as in “hey, my uncle posted an AI generated video of Biden eating a baby on FB”, no (though that doesn’t help our readiness as a species). The general ability of AI to deceive, impersonate, and pretty much break any assumption about who we may believe we are talking to though is a prominent detail that often features in extinction scenarios (e.g. how the AI starts making its own money or manipulating humans into producing thing it needs).
I would say “extinction scenarios” include everything that features extinction and AI in the event chain, which doesn’t even strictly need to be a takeover by agentic AI. Anyway the actual signed statement is very general. I can guess that some of these people don’t worry specifically about the “you are made of atoms” scenario, but that’s just arguing against something that isn’t in the statement.
My point is I don’t think they’re incorrect. All those things are ALSO problems, and many are paths to X-risk even, which I’d consider more likely (in a slow takeoff scenario) than FOOM. A few possible scenarios:
designer pathogens are the obvious example because there’s so many ways they can cause targeted human extinction without risking the AI’s integrity in any way, so they’re obvious candidates for both a misuse of AI by malicious humans and for a rogue AI that is actively trying to kill us for whatever reason. Plus there’s also the related risk of designer microorganisms that alter the Earth’s biosphere as a whole, which would be even deadlier (think something like the cyanobacteria that caused the Great Oxidation Event, except now it’d be something that grows out of control and makes the atmosphere deadly)
autonomous weapons are obviously dangerous in the hands of an AI because that’s really handing it our defenses on a silver platter and makes it turning against us much more likely to succeed (and thus a potentially attractive strategy to seek power)
deepfakes or actual even more sophisticated forms of deception could be exploited by an AI to manipulate humans towards carrying out actions that benefit it or allow it to escape confinement and so on. Being able to see through deepfakes and defend against them would be key to the security of important strategic resources
we don’t know what happens socially and economically if AGI really takes everyone’s job. We go into unexplored territory, and more so, we go there from a place where the AGI will probably be owned by a few to begin with. We might hope for glorious post-scarcity abundance but that’s not the only road nor, I fear, the most likely. If the transition goes badly it can weaken our species enough that all AGI needs to do to get rid of us for good is give a little push.
All these things of course are only relevant with relatively weak-ish AGI. If it is possible to have a self-improving AGI that FOOMs in a matter of hours and kills everyone with nanobots, then we’re screwed either way. But we don’t know that’s possible, the slower scenario are also possible and we can at least defend more against them. Other strategies, like “find ways to align the AGI” or “just don’t build the damn thing” prevent all the scenarios anyway. So I don’t think this statement is particularly missing the mark. Remember that the narrower the statement, the less people can endorse it. This is the Minimum Common Denominator: the things that all the signatories, even with probably very different individual viewpoints, can agree for sure are an issue. Which they are. There’s probably more issues, but consensus on those is not as solid.
Misconstruing an incorrect statement with a correct steelman is incorrect. If I say “I’ve discovered a truly marvelous proof that 2+2=3000 that this margin is too small to contain,” and you reply, “Ah, so you are saying 2+2=4, quite right,” then the fact of your inexplicable discussion of a different and correct statement doesn’t make your interpretation of my original incorrect statement correct.
I explained in the rest of the comment why I don’t think they’re incorrect, literally. The signed statement anyway is just:
So that seems to me like it’s succinct enough to include all categories discussed here. I don’t see the issue.
“Extinction from AI” really doesn’t refer to deepfakes, AI leaving us nothing to do, and algorithmic bias. It doesn’t include any of these categories. There is nothing correct about interpreting “extinction from AI” as referring to either of those things. This holds even if extinction from AI is absolutely impossible, and those other things are both real/imminent and extremely important. Words have meaning even when they refer to fictional ideas.
Deepfakes as in “hey, my uncle posted an AI generated video of Biden eating a baby on FB”, no (though that doesn’t help our readiness as a species). The general ability of AI to deceive, impersonate, and pretty much break any assumption about who we may believe we are talking to though is a prominent detail that often features in extinction scenarios (e.g. how the AI starts making its own money or manipulating humans into producing thing it needs).
I would say “extinction scenarios” include everything that features extinction and AI in the event chain, which doesn’t even strictly need to be a takeover by agentic AI. Anyway the actual signed statement is very general. I can guess that some of these people don’t worry specifically about the “you are made of atoms” scenario, but that’s just arguing against something that isn’t in the statement.