“One day, one of the AGI systems improves to the point where it unlocks a new technology that can reliably kill all humans, as well as destroying all of its AGI rivals. (E.g., molecular nanotechnology.) I predict that regardless of how well-behaved it’s been up to that point, it uses the technology and takes over. Do you predict otherwise?”
I agree with this, given your assumptions. But this seems like a fast take off scenario, right? My main question wasn’t addressed — are we assuming a fast take off? I didn’t see that explicitly discussed.
My understanding is that common law isn’t easy to change, even if individual agents would prefer to. This is why there are Nash equilibria. Of course, if there’s a fast enough take off, then this is irrelevant.
I would define hard takeoff as “progress in cognitive ability from pretty-low-impact AI to astronomically high-impact AI is discontinuous, and fast in absolute terms”.
Unlocking a technology that lets you kill other powerful optimizers (e.g., nanotech) doesn’t necessarily require fast or discontinuous improvements to systems’ cognition. E.g., humans invented nuclear weapons just via accumulating knowledge over time; the invention wasn’t caused by us surgically editing the human brain a few years prior to improve its reasoning. (Though software improvements like ‘use scientific reasoning’, centuries prior, were obviously necessary.)
“One day, one of the AGI systems improves to the point where it unlocks a new technology that can reliably kill all humans, as well as destroying all of its AGI rivals. (E.g., molecular nanotechnology.) I predict that regardless of how well-behaved it’s been up to that point, it uses the technology and takes over. Do you predict otherwise?”
I agree with this, given your assumptions. But this seems like a fast take off scenario, right? My main question wasn’t addressed — are we assuming a fast take off? I didn’t see that explicitly discussed.
My understanding is that common law isn’t easy to change, even if individual agents would prefer to. This is why there are Nash equilibria. Of course, if there’s a fast enough take off, then this is irrelevant.
I would define hard takeoff as “progress in cognitive ability from pretty-low-impact AI to astronomically high-impact AI is discontinuous, and fast in absolute terms”.
Unlocking a technology that lets you kill other powerful optimizers (e.g., nanotech) doesn’t necessarily require fast or discontinuous improvements to systems’ cognition. E.g., humans invented nuclear weapons just via accumulating knowledge over time; the invention wasn’t caused by us surgically editing the human brain a few years prior to improve its reasoning. (Though software improvements like ‘use scientific reasoning’, centuries prior, were obviously necessary.)