I’m not sure what you mean: I expect that given any conflict scenario which has a potential for nuclear war, there is some outcome which ~all powerful people prefer to a nuclear war, even if their preferences are quite opposed, so why do you expect that building AI which enhances the ability of the powerful to achieve their preferences, including through negotiation, would lead to nuclear war?
Traditionally, nuclear war is predicted to be a result of incompetence, or of breakdown of negotiations, rather than malice, and I’m not sure why b) would make those things more likely.
ASI tends to introduce a strong “winner takes all” dynamic. At some point, the people who aren’t winning, but still have nukes and haven’t yet lost so decisively they can no longer use them, may decide they’re better off expressing their displeasure with a limited nuclear exchange — which may not stay limited. IMO, the combination of high-stakes winner-take-all competition and weapons of mass destruction is a volatile one, and that’s where b) leads. But I’m trying to peer past a singularity here: an ASI might find an easy counter that I’m not seeing.
Basically, I trust human morality — it’s a very heavily battle-tested system for finding acceptable compromises between conflicting parties and encouraging them to achieve mutually beneficial cooperation. So I think we’re better off with it than without it. I’m hoping even a type b) ASI would agree. But as I said, I might just be an idealist.
I’m not sure what you mean: I expect that given any conflict scenario which has a potential for nuclear war, there is some outcome which ~all powerful people prefer to a nuclear war, even if their preferences are quite opposed, so why do you expect that building AI which enhances the ability of the powerful to achieve their preferences, including through negotiation, would lead to nuclear war?
Traditionally, nuclear war is predicted to be a result of incompetence, or of breakdown of negotiations, rather than malice, and I’m not sure why b) would make those things more likely.
ASI tends to introduce a strong “winner takes all” dynamic. At some point, the people who aren’t winning, but still have nukes and haven’t yet lost so decisively they can no longer use them, may decide they’re better off expressing their displeasure with a limited nuclear exchange — which may not stay limited. IMO, the combination of high-stakes winner-take-all competition and weapons of mass destruction is a volatile one, and that’s where b) leads. But I’m trying to peer past a singularity here: an ASI might find an easy counter that I’m not seeing.
Basically, I trust human morality — it’s a very heavily battle-tested system for finding acceptable compromises between conflicting parties and encouraging them to achieve mutually beneficial cooperation. So I think we’re better off with it than without it. I’m hoping even a type b) ASI would agree. But as I said, I might just be an idealist.