I think that Eliezer thinks p(doom)> 99% and many others here are following in his wake. He is making a lot of speculative inferences. But even allowing for that, and rejecting instrumental convergence, p(doom) is uncomfortably high (though probably not greater than 99%).
You think that it is wrong to say: (i) in 10-20 years there will be (ii) a single AI (iii) that will kill all humans quickly (iv) before we can respond.
Eliezer is not saying point ii. He certainly seems to think there could be multiple AIs. (It doesn’t make a difference, as far as I can tell, whether human extinction occurs at the hands of one or many AIs. You can argue that the existence of multiple AIs will retard human extinction, but you could equally argue that it would speed human extinction. Both arguments are speculation without evidence and shouldn’t change estimates of p(doom).)
I don’t think we will have AGI in 10-20 years. But, if I’m putting guesstimated probabilities on this, I can’t say the chance is less than 10% after 10 years. To put it another way, 10 years from now there is a realistic chance that we will have AGI, even if it isn’t likely. And, if you believe that conditional on having AGI, p(doom) is very high, that is the time frame you care about, because that is when you need to have a way for humanity to prevent this catastrophe (if possible).
It’s better if it takes 30 or 50 years. But, he doesn’t see that we are likely to have a realistic implementable plan to prevent human extinction then either (and neither do I, FWIW). And, unless you think that we will be able to deal with AGI then in a way we can’t now, it doesn’t make a difference to humanity whether the timeline is 10-20 years or 60-80.
In other words, you may be right about point i, but it doesn’t matter.
What really matters are points iii and iv. With regard to point iv, an AGI will have an OODA loop that is faster than humans do. It’s almost definitional. It will be an entity rather than an organization, it will be smarter and it will think faster.
That leaves point iii which breaks down into whether the AGI will kill humanity and whether it can. If the AGI can kill most of humanity and intends to kill all of humanity it will be able to do so. By killing most of humanity, the ability of humanity to fight back will be crippled. You think that the AGI can kill large numbers of people; I’m not sure whether you think it can kill most of humanity, but without appeals to technology substantially in advance of our own I can think of multiple ways for an AGI to achieve this. (Pandemics with viruses designed to be 100% lethal and highly transmissible. Nuclear holocaust. Push climate change into overdrive. Robots to hunt and kill surviving humans. )
Will the AGI decide to kill us all? I think the answer here is maybe while Eliezer seems confident it is yes.
I agree that points iii and IV are the relevant ones. Just to clarify, no, I don’t think it can’t kill most of humanity and I think that people thinking that they can come up with valid plans themselves (and by extension an AGI could too) are overestimating the things that can be known/predicted/plan in a highly complex system. I do think it can kill millions of humans though, but this is not what is being said. I think that what is being said is alarmist, and that it will have a cost eventually.
Civilization is a highly complex and fragile system, without with most of humanity will die and humanity will be rendered defenseless. If you want to destroy it, you don’t have to predict or plan what will happen, you just have to hit it hard and fast, preferably from a couple of different directions.
There is an implicit norm here against provided detailed plans to destroy civilization so I won’t, but it is not hard to come up with one (or four) and you will likely have thought of some yourself. The key thing is that if you get to hit again (and the AGI will) you only need to achieve a portion of your objective with each try.
The problem is that you not only have to hit hard and first, you have to prevent any possible retaliation because you hitting means that run the risk of being yourself hit. Are you telling me that you can conceive different ways to derail humanity but you can’t imagine a machine concluding that the risk is too high to play that game?
I can certainly imagine a machine concluding that the risk is too high to want to play that game. And I can imagine other reasons a machine might decide not to end humanity. That is why I wind up at maybe instead of definitely (i.e. p(doom) < 99%).
But that ultimately becomes a question of the machine’s goals, motivation, understanding, agency and risk tolerance. I think that there is a wide distribution of these and therefore an unknown but significant chance that the AGI decides not to destroy humanity.
That is very different from the question of whether the AGI could achieve the destruction of humanity. If the AGI couldn’t destroy humanity in practice, p(doom) would be close to 0.
In other words, I think the AGI can kill humanity but may choose not to. You seemed above to think the AGI can’t, but now seem to think it might be able to but may choose not to.
I think that Eliezer thinks p(doom)> 99% and many others here are following in his wake. He is making a lot of speculative inferences. But even allowing for that, and rejecting instrumental convergence, p(doom) is uncomfortably high (though probably not greater than 99%).
You think that it is wrong to say: (i) in 10-20 years there will be (ii) a single AI (iii) that will kill all humans quickly (iv) before we can respond.
Eliezer is not saying point ii. He certainly seems to think there could be multiple AIs. (It doesn’t make a difference, as far as I can tell, whether human extinction occurs at the hands of one or many AIs. You can argue that the existence of multiple AIs will retard human extinction, but you could equally argue that it would speed human extinction. Both arguments are speculation without evidence and shouldn’t change estimates of p(doom).)
I don’t think we will have AGI in 10-20 years. But, if I’m putting guesstimated probabilities on this, I can’t say the chance is less than 10% after 10 years. To put it another way, 10 years from now there is a realistic chance that we will have AGI, even if it isn’t likely. And, if you believe that conditional on having AGI, p(doom) is very high, that is the time frame you care about, because that is when you need to have a way for humanity to prevent this catastrophe (if possible).
It’s better if it takes 30 or 50 years. But, he doesn’t see that we are likely to have a realistic implementable plan to prevent human extinction then either (and neither do I, FWIW). And, unless you think that we will be able to deal with AGI then in a way we can’t now, it doesn’t make a difference to humanity whether the timeline is 10-20 years or 60-80.
In other words, you may be right about point i, but it doesn’t matter.
What really matters are points iii and iv. With regard to point iv, an AGI will have an OODA loop that is faster than humans do. It’s almost definitional. It will be an entity rather than an organization, it will be smarter and it will think faster.
That leaves point iii which breaks down into whether the AGI will kill humanity and whether it can. If the AGI can kill most of humanity and intends to kill all of humanity it will be able to do so. By killing most of humanity, the ability of humanity to fight back will be crippled. You think that the AGI can kill large numbers of people; I’m not sure whether you think it can kill most of humanity, but without appeals to technology substantially in advance of our own I can think of multiple ways for an AGI to achieve this. (Pandemics with viruses designed to be 100% lethal and highly transmissible. Nuclear holocaust. Push climate change into overdrive. Robots to hunt and kill surviving humans. )
Will the AGI decide to kill us all? I think the answer here is maybe while Eliezer seems confident it is yes.
I agree that points iii and IV are the relevant ones. Just to clarify, no, I don’t think it can’t kill most of humanity and I think that people thinking that they can come up with valid plans themselves (and by extension an AGI could too) are overestimating the things that can be known/predicted/plan in a highly complex system. I do think it can kill millions of humans though, but this is not what is being said. I think that what is being said is alarmist, and that it will have a cost eventually.
Civilization is a highly complex and fragile system, without with most of humanity will die and humanity will be rendered defenseless. If you want to destroy it, you don’t have to predict or plan what will happen, you just have to hit it hard and fast, preferably from a couple of different directions.
There is an implicit norm here against provided detailed plans to destroy civilization so I won’t, but it is not hard to come up with one (or four) and you will likely have thought of some yourself. The key thing is that if you get to hit again (and the AGI will) you only need to achieve a portion of your objective with each try.
The problem is that you not only have to hit hard and first, you have to prevent any possible retaliation because you hitting means that run the risk of being yourself hit. Are you telling me that you can conceive different ways to derail humanity but you can’t imagine a machine concluding that the risk is too high to play that game?
I can certainly imagine a machine concluding that the risk is too high to want to play that game. And I can imagine other reasons a machine might decide not to end humanity. That is why I wind up at maybe instead of definitely (i.e. p(doom) < 99%).
But that ultimately becomes a question of the machine’s goals, motivation, understanding, agency and risk tolerance. I think that there is a wide distribution of these and therefore an unknown but significant chance that the AGI decides not to destroy humanity.
That is very different from the question of whether the AGI could achieve the destruction of humanity. If the AGI couldn’t destroy humanity in practice, p(doom) would be close to 0.
In other words, I think the AGI can kill humanity but may choose not to. You seemed above to think the AGI can’t, but now seem to think it might be able to but may choose not to.