Define an existential catastrophe due to AI as an existential catastrophe that could have been avoided had humanity’s development, deployment or governance of AI been otherwise. This includes cases where:
AI directly causes the catastrophe.
AI is a significant risk factor in the catastrophe, such that no catastrophe would have occurred without the involvement of AI.
Humanity survives but its suboptimal use of AI means that we fall permanently and drastically short of our full potential.
This technically seems to include cases like: AGI is not developed by 2050, and a nuclear war in the year 2050 causes an existential catastrophe, but if an aligned AGI had been developed by then, it would have prevented the nuclear war. I don’t know if respondents interpreted it that way.
Thanks for pointing this out. We did intend for cases like this to be included, but I agree that it’s unclear if respondents interpreted it that way. We should have clarified this in the survey instructions.
Thanks for your comment! I think your critique is justified.
My best guess is that this consideration was not salient for most participants and probably didn’t distort the results in meaningful ways, but it’s of course hard to tell and DanielFilan’s comment suggests that it was not irrelevant.
We are aware of a number of other limitations, especially with regards to the mutual exclusivity of different scenarios. We’ve summarized these limitations here.
Overall, you should take the results with a grain of salt. They should only be seen as signposts indicating which scenarios people find most plausible.
Is one question combining the risk of “too much” AI use and “too little” AI use?
Yes, it is. Combining these cases seems reasonable to me, though we definitely should have clarified this in the survey instructions. They’re both cases where humanity could avoided an existential catastrophe by making different decisions with respect to AI.
But the action needed to avoid/mitigate in those cases is very different, so it doesn’t seem useful to get a feeling for “how far off of ideal are we likely to be” when that is composed of: 1. What is the possible range of AI functionality (as constrained by physics)? - ie what can we do?
2. What is the range of desirable outcomes within that range? - ie what should we do?
3. How will politics, incumbent interests, etc. play out? - ie what will we actually do?
Knowing that experts think we have a (say) 10% chance of hitting the ideal window says nothing about what an interested party should do to improve those chances. It could be “attempt to shut down all AI research” or “put more funding into AI research” or “it doesn’t matter because the two majority cases are “General AI is impossible − 40%” and “General AI is inevitable and will wreck us − 50%”″
it doesn’t seem useful to get a feeling for “how far off of ideal are we likely to be” when that is composed of: 1. What is the possible range of AI functionality (as constrained by physics)? - ie what can we do?
No, these cases aren’t included. The definition is: “an existential catastrophe that could have been avoided had humanity’s development, deployment or governance of AI been otherwise”. Physics cannot be changed by humanity’s development/deployment/governance decisions. (I agree that cases 2 and 3 are included).
Knowing that experts think we have a (say) 10% chance of hitting the ideal window says nothing about what an interested party should do to improve those chances.
That’s correct. The survey wasn’t intended to understand respondents’ views on interventions. It was only intended to understand: if something goes wrong, what do respondents think that was? Someone could run another survey that asks about interventions (in fact, this other recent survey does that). For the reasons given in the Motivation section of this post, we chose to limit our scope to threat models, rather than interventions.
This technically seems to include cases like: AGI is not developed by 2050, and a nuclear war in the year 2050 causes an existential catastrophe, but if an aligned AGI had been developed by then, it would have prevented the nuclear war. I don’t know if respondents interpreted it that way.
Thanks for pointing this out. We did intend for cases like this to be included, but I agree that it’s unclear if respondents interpreted it that way. We should have clarified this in the survey instructions.
Thanks for your comment! I think your critique is justified.
My best guess is that this consideration was not salient for most participants and probably didn’t distort the results in meaningful ways, but it’s of course hard to tell and DanielFilan’s comment suggests that it was not irrelevant.
We are aware of a number of other limitations, especially with regards to the mutual exclusivity of different scenarios. We’ve summarized these limitations here.
Overall, you should take the results with a grain of salt. They should only be seen as signposts indicating which scenarios people find most plausible.
As a respondent, I remember being unsure whether I should include those catastrophes.
That seems like a really bad conflation? Is one question combining the risk of “too much” AI use and “too little” AI use?
That’s even worse than the already widely smashed distinctions between “can we?” “should we?” And “will we?”
Yes, it is. Combining these cases seems reasonable to me, though we definitely should have clarified this in the survey instructions. They’re both cases where humanity could avoided an existential catastrophe by making different decisions with respect to AI.
But the action needed to avoid/mitigate in those cases is very different, so it doesn’t seem useful to get a feeling for “how far off of ideal are we likely to be” when that is composed of:
1. What is the possible range of AI functionality (as constrained by physics)? - ie what can we do?
2. What is the range of desirable outcomes within that range? - ie what should we do?
3. How will politics, incumbent interests, etc. play out? - ie what will we actually do?
Knowing that experts think we have a (say) 10% chance of hitting the ideal window says nothing about what an interested party should do to improve those chances. It could be “attempt to shut down all AI research” or “put more funding into AI research” or “it doesn’t matter because the two majority cases are “General AI is impossible − 40%” and “General AI is inevitable and will wreck us − 50%”″
Thanks for the reply—a couple of responses:
No, these cases aren’t included. The definition is: “an existential catastrophe that could have been avoided had humanity’s development, deployment or governance of AI been otherwise”. Physics cannot be changed by humanity’s development/deployment/governance decisions. (I agree that cases 2 and 3 are included).
That’s correct. The survey wasn’t intended to understand respondents’ views on interventions. It was only intended to understand: if something goes wrong, what do respondents think that was? Someone could run another survey that asks about interventions (in fact, this other recent survey does that). For the reasons given in the Motivation section of this post, we chose to limit our scope to threat models, rather than interventions.