I’ve made a distribution based on the Metaculus community distributions:
(I used this Colab notebook for generating the plots from Elicit distributions over specific risks. My Elicit snapshot is here).
In 2019, Metaculus posted the results of a forecasting series on catastrophic risk (>95% of humans die) by 2100. The overall risk was 9.2% for the community forecast (with 7.3% for AI risk). To convert this to a forecast for existential risk (100% dead), I assumed 6% risk from AI, 1% from nuclear war, and 0.4% from biological risk. To get timelines, I used Metaculus forecasts for when the AI catastrophe occurs and for when great power war happens (as a rough proxy for nuclear war). I put my own uninformative distribution on biological risk.
This shouldn’t be taken as the “Metaculus” forecast, as I’ve made various extrapolations. Moreover, Metaculus has a separate question about x-risk, where the current forecast is 2% by 2100. This seems to me hard to reconcile with the 7% chance of AI killing >95% of people by 2100, and so I’ve used the latter as my source.
Technical note: I normalized the timeline pdfs based on the Metaculus binary probabilities in this table, and then treated them as independent sources of x-risk using the Colab. This inflates the overall x-risk slightly. However, this could be fixed by re-scaling the cdfs.
Very interesting, thanks for sharing! This seems like a nice example of combining various existing predictions to answer a new question.
a forecast for existential risk (100% dead)
It seems worth highlighting that extinction risk (risk of 100% dead) is a (big) subset of existential risk (risk of permanent and drastic destruction of humanity’s potential), rather than those two terms being synonymous. If your forecast was for extinction risk only, then the total existential risk should presumably be at least slightly higher, due to risks of unrecoverable collapse or unrecoverable dystopia.
(I think it’s totally ok and very useful to “just” forecast extinction risk. I just think it’s also good to be clear about what one’s forecast is of.)
Good points. Unfortunately it seems even harder to infer “destruction of potential” from the Metaculus forecasts. It seems plausible that AI could cause destruction of potential without any deaths at all, and so this wouldn’t be covered by the Metaculus series.
I also think it’s easier to forecast extinction in general, partly because it’s a much clearer threshold, whereas there are some scenarios that some people might count as an “existential catastrophe” and others might not. (E.g., Bostrom’s “plateauing — progress flattens out at a level perhaps somewhat higher than the present level but far below technological maturity”.)
The overall risk was 9.2% for the community forecast (with 7.3% for AI risk). To convert this to a forecast for existential risk (100% dead), I assumed 6% risk from AI, 1% from nuclear war, and 0.4% from biological risk
I think this implies you think:
AI is ~4 or 5 times (6% vs 1.3%) as likely to kill 100% of people as to kill between 95 and 100% of people
Everything other than AI is roughly equally likely (1.5% vs 1.4%) to kill 100% of people as to kill between 95% and 100% of people
Does that sound right to you? And if so, what was your reasoning?
I ask out of curiosity, not because I disagree. I don’t have a strong view here, except perhaps that AI is the risk with the highest ratio of “chance it causes outright extinction” to “chance it causes major carnage” (and this seems to align with your views).
The Metaculus community forecast has chance of >95% dead (7.5%) close to chance of >10% dead (9.7%) for AI. Based on this and my own intuition about how AI risks “scale”, I extrapolated to 6% for 100% dead. For biological and nuclear war, there’s a much bigger drop off from >10% to >95% from the community. It’s hard to say what to infer from this about the 100% case. There are good arguments that 100% is unlikely from both, but some of those arguments would also cut against >95%. I didn’t do a careful examination and so take all these numbers with a grain of salt.
I’ve made a distribution based on the Metaculus community distributions:
(I used this Colab notebook for generating the plots from Elicit distributions over specific risks. My Elicit snapshot is here).
In 2019, Metaculus posted the results of a forecasting series on catastrophic risk (>95% of humans die) by 2100. The overall risk was 9.2% for the community forecast (with 7.3% for AI risk). To convert this to a forecast for existential risk (100% dead), I assumed 6% risk from AI, 1% from nuclear war, and 0.4% from biological risk. To get timelines, I used Metaculus forecasts for when the AI catastrophe occurs and for when great power war happens (as a rough proxy for nuclear war). I put my own uninformative distribution on biological risk.
This shouldn’t be taken as the “Metaculus” forecast, as I’ve made various extrapolations. Moreover, Metaculus has a separate question about x-risk, where the current forecast is 2% by 2100. This seems to me hard to reconcile with the 7% chance of AI killing >95% of people by 2100, and so I’ve used the latter as my source.
Technical note: I normalized the timeline pdfs based on the Metaculus binary probabilities in this table, and then treated them as independent sources of x-risk using the Colab. This inflates the overall x-risk slightly. However, this could be fixed by re-scaling the cdfs.
Very interesting, thanks for sharing! This seems like a nice example of combining various existing predictions to answer a new question.
It seems worth highlighting that extinction risk (risk of 100% dead) is a (big) subset of existential risk (risk of permanent and drastic destruction of humanity’s potential), rather than those two terms being synonymous. If your forecast was for extinction risk only, then the total existential risk should presumably be at least slightly higher, due to risks of unrecoverable collapse or unrecoverable dystopia.
(I think it’s totally ok and very useful to “just” forecast extinction risk. I just think it’s also good to be clear about what one’s forecast is of.)
Good points. Unfortunately it seems even harder to infer “destruction of potential” from the Metaculus forecasts. It seems plausible that AI could cause destruction of potential without any deaths at all, and so this wouldn’t be covered by the Metaculus series.
Yeah, totally agreed.
I also think it’s easier to forecast extinction in general, partly because it’s a much clearer threshold, whereas there are some scenarios that some people might count as an “existential catastrophe” and others might not. (E.g., Bostrom’s “plateauing — progress flattens out at a level perhaps somewhat higher than the present level but far below technological maturity”.)
I think this implies you think:
AI is ~4 or 5 times (6% vs 1.3%) as likely to kill 100% of people as to kill between 95 and 100% of people
Everything other than AI is roughly equally likely (1.5% vs 1.4%) to kill 100% of people as to kill between 95% and 100% of people
Does that sound right to you? And if so, what was your reasoning?
I ask out of curiosity, not because I disagree. I don’t have a strong view here, except perhaps that AI is the risk with the highest ratio of “chance it causes outright extinction” to “chance it causes major carnage” (and this seems to align with your views).
The Metaculus community forecast has chance of >95% dead (7.5%) close to chance of >10% dead (9.7%) for AI. Based on this and my own intuition about how AI risks “scale”, I extrapolated to 6% for 100% dead. For biological and nuclear war, there’s a much bigger drop off from >10% to >95% from the community. It’s hard to say what to infer from this about the 100% case. There are good arguments that 100% is unlikely from both, but some of those arguments would also cut against >95%. I didn’t do a careful examination and so take all these numbers with a grain of salt.