ASI tends to introduce a strong “winner takes all” dynamic. At some point, the people who aren’t winning, but still have nukes and haven’t yet lost so decisively they can no longer use them, may decide they’re better off expressing their displeasure with a limited nuclear exchange — which may not stay limited. IMO, the combination of high-stakes winner-take-all competition and weapons of mass destruction is a volatile one, and that’s where b) leads. But I’m trying to peer past a singularity here: an ASI might find an easy counter that I’m not seeing.
Basically, I trust human morality — it’s a very heavily battle-tested system for finding acceptable compromises between conflicting parties and encouraging them to achieve mutually beneficial cooperation. So I think we’re better off with it than without it. I’m hoping even a type b) ASI would agree. But as I said, I might just be an idealist.
ASI tends to introduce a strong “winner takes all” dynamic. At some point, the people who aren’t winning, but still have nukes and haven’t yet lost so decisively they can no longer use them, may decide they’re better off expressing their displeasure with a limited nuclear exchange — which may not stay limited. IMO, the combination of high-stakes winner-take-all competition and weapons of mass destruction is a volatile one, and that’s where b) leads. But I’m trying to peer past a singularity here: an ASI might find an easy counter that I’m not seeing.
Basically, I trust human morality — it’s a very heavily battle-tested system for finding acceptable compromises between conflicting parties and encouraging them to achieve mutually beneficial cooperation. So I think we’re better off with it than without it. I’m hoping even a type b) ASI would agree. But as I said, I might just be an idealist.