To restate what other people have said- the uncertainty is with the assumptions, not the nature of the world that would result if the assumptions were true.
To analogize- it’s like we’re imagining a massive complex bomb could exist in the future made out of a hypothesized highly reactive chemical.
The uncertainty that influences p(DOOM) isn’t ‘maybe the bomb will actually be very easy to defuse,’ or ‘maybe nobody will touch the bomb and we can just leave it there,’ it’s ‘maybe the chemical isn’t manufacturable,’ ‘maybe the chemical couldn’t be stored in the first place,’ or ‘maybe the chemical just wouldn’t be reactive at all.’
So to transfer back from the analogy, you are saying the uncertainty is in “maybe it’s not possible to create a God-like AI” and “maybe people won’t create a God-like AI” and “maybe a God-like AI won’t do anything”?
Another one, corresponding to the analogy in the chemical not being reactive at all, is the possbility that even very strong AIs are fundamentally very easy to align by default, for any number of reasons.
To restate what other people have said- the uncertainty is with the assumptions, not the nature of the world that would result if the assumptions were true.
To analogize- it’s like we’re imagining a massive complex bomb could exist in the future made out of a hypothesized highly reactive chemical.
The uncertainty that influences p(DOOM) isn’t ‘maybe the bomb will actually be very easy to defuse,’ or ‘maybe nobody will touch the bomb and we can just leave it there,’ it’s ‘maybe the chemical isn’t manufacturable,’ ‘maybe the chemical couldn’t be stored in the first place,’ or ‘maybe the chemical just wouldn’t be reactive at all.’
So to transfer back from the analogy, you are saying the uncertainty is in “maybe it’s not possible to create a God-like AI” and “maybe people won’t create a God-like AI” and “maybe a God-like AI won’t do anything”?
Another one, corresponding to the analogy in the chemical not being reactive at all, is the possbility that even very strong AIs are fundamentally very easy to align by default, for any number of reasons.