The cruxes you have picked out are not the ones I would have.
The complete argument for complete extinction rests on assumptions...assumptions about the nature of intelligence, the motivations of an artificial intelligence and the means of bringing about extinction. And the conjunctive part of the argument consists of claims which need to be of high probability individually for the conclusion to be of high probability.
Artificial Intelligence greater than human intelligence is possible.
The AI will be an agent, it have goals/values in the first place.
The goals will be misaligned, however subtly, to be unfavorable to humanity.
That the misalignment between the AI’s goals, and what we want, cannot be corrected incrementally (incorrigibility), because
5a. …the AI will self modify in way too fast to stop.(With a sub assumption that the AI can achieve value stability under self modification.)
Or
5b. …the AI will.The engage in deception about its powers or motivations.
That most misaligned values in the resulting ASI are highly dangerous (even goals that aren’t directly inimical to humans can be a problem for humans)
And that the AI will have extensive opportunities to wreak havoc: biological warfare (custom DNA can be ordered by email), crashing economic systems (trading can be done online), taking over weapon systems, weaponing other technology and so on.
Obviously the problem is that to claim a high overall probability of doom, each claim in the chain needs to have a high probability. It is not enough for some of the stages to be highly probable, all must be. In my opinion, the weakest parts of the argument are the ones dealing with the motivation, steps 2 to 6, not the ones dealing with the natures of intelligence and the means of destruction, (1 and 7). There’s an obvious problem in making specific high probability claims about systems that don’t yet exist.
It’s kind of strange. Why do we want to have a technical ability for any Mr.X from the defense department of a superpower Y to impart his goals and values to some ASI? It’s very easy to imagine how this could be detrimental
Yes, but not everybody’s-dead detrimental.
Doomers are concerned about imparting values because they believe that we are going to end up with an incorrigible Sovereign AI running everything, not a multipolar scenario with superpowers using aligned-with-themselves superintelligences as superweapons.
The cruxes you have picked out are not the ones I would have.
The complete argument for complete extinction rests on assumptions...assumptions about the nature of intelligence, the motivations of an artificial intelligence and the means of bringing about extinction. And the conjunctive part of the argument consists of claims which need to be of high probability individually for the conclusion to be of high probability.
Artificial Intelligence greater than human intelligence is possible.
The AI will be an agent, it have goals/values in the first place.
The goals will be misaligned, however subtly, to be unfavorable to humanity.
That the misalignment between the AI’s goals, and what we want, cannot be corrected incrementally (incorrigibility), because
5a. …the AI will self modify in way too fast to stop.(With a sub assumption that the AI can achieve value stability under self modification.)
Or
5b. …the AI will.The engage in deception about its powers or motivations.
That most misaligned values in the resulting ASI are highly dangerous (even goals that aren’t directly inimical to humans can be a problem for humans)
And that the AI will have extensive opportunities to wreak havoc: biological warfare (custom DNA can be ordered by email), crashing economic systems (trading can be done online), taking over weapon systems, weaponing other technology and so on.
Obviously the problem is that to claim a high overall probability of doom, each claim in the chain needs to have a high probability. It is not enough for some of the stages to be highly probable, all must be. In my opinion, the weakest parts of the argument are the ones dealing with the motivation, steps 2 to 6, not the ones dealing with the natures of intelligence and the means of destruction, (1 and 7). There’s an obvious problem in making specific high probability claims about systems that don’t yet exist.
@Mishka
Yes, but not everybody’s-dead detrimental.
Doomers are concerned about imparting values because they believe that we are going to end up with an incorrigible Sovereign AI running everything, not a multipolar scenario with superpowers using aligned-with-themselves superintelligences as superweapons.