In the OP, I contrasted the “comparatively-less-pessimistic group (say, P(doom)…in the 5%–50% range”, with the “even more pessimistic group” that includes me. I think the style of argument you mention (“strong economic and military incentives for making minds that are highly agent-y…”) is a good argument for being in the former group, but it’s kinda hard to justify P(doom)>>50% that way.
For example, an optimist could respond to the “incentives” argument by saying “well humans can be pretty agent-y, and humans can accomplish ambitious projects, but yet humans also care about our friends and follow local norms and customs etc. So it’s at least possible that AIs could be like that!” And then the pessimist could respond “yeah but the more ruthless AIs will outcompete the nice ones”, and then the optimist could respond “well the vast majority of the AIs will be nice because competent companies and militaries won’t choose to run AIs that are eager to wipe out humanity, so the few omnicidal AIs can be stopped”, and then maybe the pessimist would pivot to offense-defense balance or whatever, and on we go down the argument tree. Anyway, there are some people who get to P(doom)>>50% via these kinds of arguments, but I think it’s more common that these arguments only get people into the P(doom)≈5–50% range.
And then I’m in a different cluster that includes Eliezer & Nate, and says that “P(doom)>>50% because egregious omnicidal misalignment is what’s definitely going to happen in the absence of some unlikely technical breakthrough”. That’s what I was defending in this post.
For many purposes, the “comparatively-less-pessimistic group” and the “even more pessimistic group” are on the same team, e.g. against Marc Andreesen. But in other contexts, the two groups are on opposite sides, so it’s very worthwhile to try to hash out which group is right.
In the OP, I contrasted the “comparatively-less-pessimistic group (say, P(doom)…in the 5%–50% range”, with the “even more pessimistic group” that includes me. I think the style of argument you mention (“strong economic and military incentives for making minds that are highly agent-y…”) is a good argument for being in the former group, but it’s kinda hard to justify P(doom)>>50% that way.
For example, an optimist could respond to the “incentives” argument by saying “well humans can be pretty agent-y, and humans can accomplish ambitious projects, but yet humans also care about our friends and follow local norms and customs etc. So it’s at least possible that AIs could be like that!” And then the pessimist could respond “yeah but the more ruthless AIs will outcompete the nice ones”, and then the optimist could respond “well the vast majority of the AIs will be nice because competent companies and militaries won’t choose to run AIs that are eager to wipe out humanity, so the few omnicidal AIs can be stopped”, and then maybe the pessimist would pivot to offense-defense balance or whatever, and on we go down the argument tree. Anyway, there are some people who get to P(doom)>>50% via these kinds of arguments, but I think it’s more common that these arguments only get people into the P(doom)≈5–50% range.
And then I’m in a different cluster that includes Eliezer & Nate, and says that “P(doom)>>50% because egregious omnicidal misalignment is what’s definitely going to happen in the absence of some unlikely technical breakthrough”. That’s what I was defending in this post.
For many purposes, the “comparatively-less-pessimistic group” and the “even more pessimistic group” are on the same team, e.g. against Marc Andreesen. But in other contexts, the two groups are on opposite sides, so it’s very worthwhile to try to hash out which group is right.