It strikes me that there is a lot of middle ground between 1) the utopias people are trying to create with beneficial AGI, and 2) human extinction.
So even if you think #1 is no longer possible (I don’t think we’re there yet, but I know some do), I don’t think you need to leap all the way to #2 in order to address s-risk.
In a world were AGI has not yet been developed, there are surely ways to disrupt its development that don’t require building Clippy and causing human extinction. Most of these ways I wouldn’t recommend someone to pursue or feel comfortable posting on a public forum though.
That’s a fair point—my model does assume AGI will come into existence in non-negative worlds. Though I struggle to actually imagine a non-negative world where humanity is alive a thousand years from now and AGI hasn’t been developed. Even if all alignment researchers believed it was the right thing to pursue, which doesn’t seem likely.
That’s not a middle ground between a good world and a neutral world, though, that’s just another way to get a good world. If we assume a good world is exponentially unlikely, a 10 year delay might mean the odds of a good world rise from 10^-10 to 10^-8 (as opposed to pursuing Clippy bringing the odds of a bad world down from 10^-4 to 10^-6 ).
If you disagree with Yudkowsky about his pessimism about the probability of good worlds, then my post doesn’t really apply. My post is about how to handle him being correct about the odds.
It strikes me that there is a lot of middle ground between 1) the utopias people are trying to create with beneficial AGI, and 2) human extinction.
So even if you think #1 is no longer possible (I don’t think we’re there yet, but I know some do), I don’t think you need to leap all the way to #2 in order to address s-risk.
In a world were AGI has not yet been developed, there are surely ways to disrupt its development that don’t require building Clippy and causing human extinction. Most of these ways I wouldn’t recommend someone to pursue or feel comfortable posting on a public forum though.
That’s a fair point—my model does assume AGI will come into existence in non-negative worlds. Though I struggle to actually imagine a non-negative world where humanity is alive a thousand years from now and AGI hasn’t been developed. Even if all alignment researchers believed it was the right thing to pursue, which doesn’t seem likely.
Even a 5~10 year delay in AGI deployment might give enough time to solve the alignment problem.
That’s not a middle ground between a good world and a neutral world, though, that’s just another way to get a good world. If we assume a good world is exponentially unlikely, a 10 year delay might mean the odds of a good world rise from 10^-10 to 10^-8 (as opposed to pursuing Clippy bringing the odds of a bad world down from 10^-4 to 10^-6 ).
If you disagree with Yudkowsky about his pessimism about the probability of good worlds, then my post doesn’t really apply. My post is about how to handle him being correct about the odds.