I share the high concern about the potential for an existential threat from ASI, but I find the confidence levels of ≥50% X-risk in the near term to be epistemically overconfident.
To me, the probability of human extinction due to unaligned ASI must be decomposed into sequential factors.
P(ASI by 2035 while maintaining the current trajectory)≤0.75
P(no major disruption to development before 2035)≤0.8
P(alignment/control problem is NOT solved before 2035)≤0.9
P(unaligned ASI will not leave us alive for some reason)≤0.9
So 0.75×0.8×0.9×0.9≤0.486.
I am highly uncertain about these specific values and would personally prefer estimates closer to 0.5 for each component. But it seems to me that such variables should be taken into account when thinking about how doomed the world we know is.
Probabilities 1 and 2 are correlated, so you can’t multiply them.
Although, by multiplying them, you got a probability of 0.6, and the Manifold market on ASI existing by 2035 (34 traders) is trading at ~40%. So if the Manifold market is correct, then it’s true that the existential risk from ASI before 2035 is ≤0.5.
Regardless, technically the post author claimed that the probability for existential risk from ASI before 2045 was ≥0.5, and only said that it’s likely that better-than-human-AI would exist by 2035.
Well, we have surveys like this one indicating that they don’t take into account the likelihood of an existential catastrophe. It seems to me that many forecasters are thinking about a trajectory that could lead to the creation of ASI with a certain probability in a certain year. But these models can be disrupted due to other factors such as wars, sanctions, restrictions on research, social upheaval, and so on.
I share the high concern about the potential for an existential threat from ASI, but I find the confidence levels of ≥50% X-risk in the near term to be epistemically overconfident.
To me, the probability of human extinction due to unaligned ASI must be decomposed into sequential factors.
P(ASI by 2035 while maintaining the current trajectory)≤0.75
P(no major disruption to development before 2035)≤0.8
P(alignment/control problem is NOT solved before 2035)≤0.9
P(unaligned ASI will not leave us alive for some reason)≤0.9
So 0.75×0.8×0.9×0.9≤0.486.
I am highly uncertain about these specific values and would personally prefer estimates closer to 0.5 for each component. But it seems to me that such variables should be taken into account when thinking about how doomed the world we know is.
Probabilities 1 and 2 are correlated, so you can’t multiply them.
Although, by multiplying them, you got a probability of 0.6, and the Manifold market on ASI existing by 2035 (34 traders) is trading at ~40%. So if the Manifold market is correct, then it’s true that the existential risk from ASI before 2035 is ≤0.5.
Regardless, technically the post author claimed that the probability for existential risk from ASI before 2045 was ≥0.5, and only said that it’s likely that better-than-human-AI would exist by 2035.
Well, we have surveys like this one indicating that they don’t take into account the likelihood of an existential catastrophe.
It seems to me that many forecasters are thinking about a trajectory that could lead to the creation of ASI with a certain probability in a certain year. But these models can be disrupted due to other factors such as wars, sanctions, restrictions on research, social upheaval, and so on.