I think the exact quantitative details make a big difference between “AGI ruin seems nearly certain in the absense of positive miracless” and “doom seems quite plausible, but we’ll most likely make it through” (my probability of takeover is something like 35%)
I don’t think that ‘the very first STEM-level AGI is smart enough to destroy the world if you relax some precautions’ and ‘we have 2.5 years to work with STEM-level AGI before any system is smart enough to destroy the world’ changes my p(doom) much at all. (Though this is partly because I don’t expect, in either of those worlds, that we’ll be able to be confident about which world we’re in.)
If we have 6 years to safely work with STEM-level AGI, that does intuitively start to feel like a significant net increase in p(hope) to me? Though this is complicated by the fact that such AGI probably couldn’t do pivotal acts either, and having STEM-level AGI for a longer period of time before a pivotal act occurs means that the tech will be more widespread when it does reach dangerous capability levels. So in the endgame, you’re likely to have a lot more competition, and correspondingly less time to spend on safety if you want to deploy before someone destroys the world.
Thanks for the replies, Ryan!
I don’t think that ‘the very first STEM-level AGI is smart enough to destroy the world if you relax some precautions’ and ‘we have 2.5 years to work with STEM-level AGI before any system is smart enough to destroy the world’ changes my p(doom) much at all. (Though this is partly because I don’t expect, in either of those worlds, that we’ll be able to be confident about which world we’re in.)
If we have 6 years to safely work with STEM-level AGI, that does intuitively start to feel like a significant net increase in p(hope) to me? Though this is complicated by the fact that such AGI probably couldn’t do pivotal acts either, and having STEM-level AGI for a longer period of time before a pivotal act occurs means that the tech will be more widespread when it does reach dangerous capability levels. So in the endgame, you’re likely to have a lot more competition, and correspondingly less time to spend on safety if you want to deploy before someone destroys the world.