There’s two ideas mixed together here in this comment, some of which I’m planning on covering in another piece on the effects of automation on deterrence.
The first is something I’ve called the “size” of a decisive strategic advantage. Agents that primarily care about future outcomes/are extremely reconstitutable/highly risk tolerant need a much smaller technological or military lead in order to be willing to strike and secure a DSA, since they are more willing to absorb immediate damage for long-term strategic gain. Agents that value existing assets (e.g. states caring about their civilians), on the other hand, are much easier to threaten, and so need to be so strategically superior that they are not only guaranteed to win a conflict but to do so in an essentially costless way. A democratic U.S. would not try to overpower the Chinese government with new AI-enabled superweapons unless it was basically certain it could do so without suffering a catastrophic response.
This applies to AIs or states! An authoritarian country would have fewer qualms about the sacrifices they need to make for long-term security than a democratic one, nevermind a misaligned AI. It’s just a question of how easy the things the agent values are to threaten.
The second, related idea is that this creates a perverse strategic incentive to stop caring about your civilians once you’ve automated away the instrumental need to defend them. This is the military version of the gradual disempowerment story: once the industrial base is automated, and if the key decision makers are secured from consequences, it becomes easier to trade off on the security of your civilians for strategic gain. I think the primary risk here is less leaders jumping the gun to “win” a nuclear exchange (or whatever the equivalent is for future superweapons), and moreso that it will make it more attractive to do dangerous things like capture resources or negotiate under brinksmanship. Not that it’s any better to die in an accidental war over a U.S. demand for a bigger slice of lunar surface area than a direct war of conquest in mainland China, of course.
There’s two ideas mixed together here in this comment, some of which I’m planning on covering in another piece on the effects of automation on deterrence.
The first is something I’ve called the “size” of a decisive strategic advantage. Agents that primarily care about future outcomes/are extremely reconstitutable/highly risk tolerant need a much smaller technological or military lead in order to be willing to strike and secure a DSA, since they are more willing to absorb immediate damage for long-term strategic gain. Agents that value existing assets (e.g. states caring about their civilians), on the other hand, are much easier to threaten, and so need to be so strategically superior that they are not only guaranteed to win a conflict but to do so in an essentially costless way. A democratic U.S. would not try to overpower the Chinese government with new AI-enabled superweapons unless it was basically certain it could do so without suffering a catastrophic response.
This applies to AIs or states! An authoritarian country would have fewer qualms about the sacrifices they need to make for long-term security than a democratic one, nevermind a misaligned AI. It’s just a question of how easy the things the agent values are to threaten.
The second, related idea is that this creates a perverse strategic incentive to stop caring about your civilians once you’ve automated away the instrumental need to defend them. This is the military version of the gradual disempowerment story: once the industrial base is automated, and if the key decision makers are secured from consequences, it becomes easier to trade off on the security of your civilians for strategic gain. I think the primary risk here is less leaders jumping the gun to “win” a nuclear exchange (or whatever the equivalent is for future superweapons), and moreso that it will make it more attractive to do dangerous things like capture resources or negotiate under brinksmanship. Not that it’s any better to die in an accidental war over a U.S. demand for a bigger slice of lunar surface area than a direct war of conquest in mainland China, of course.
I would be interested in your piece about the effects of automation on deterrence.