The risk from AI-enabled coups in particular is detailed at length here. To reduce this risk, we can try to introduce constraints on coup-assisting uses of AI, diversify military AI suppliers, slow autocracies via export controls, and promote credible benefit-sharing.
Why are AI-enabled coups worse than the alternative in expectation? I’m personally quite uncertain about this. I looked around a bit and found an explanation from you in this podcast, which I quote below and respond to. Please let me know if you have a more detailed writeup somewhere.
People who successfully become global dictators would likely have worse values than is ideal. The process selects for Machiavellianism, sadism, and other Dark Triad traits. So that’s one thing.
I agree this has historically been the case, but think the selection pressure could be a bit different with AI-enabled global takeover, which perhaps require somewhat less political maneuvering and will to dominate other humans, and more technical foresight/understanding and appreciation of astronomical stakes. It seems unclear to me that on expectation the kind of person who becomes an AI-enabled global dictator would have worse values than average. (Of course “worse than ideal” is much more likely, but seems like the wrong standard to use here? Morality is Scary gives some explanations for my relatively low opinion of average human’s values.)
A second is that if you have an intense concentration of power, some of the mechanisms by which we get moral progress—like people being able to argue with each other and having to defend their views in the public sphere—go away.
I’m afraid that such mechanisms would probably go away even without intense concentration of power due to considerations like these. In short, AI-optimized persuasion would likely replace human debate and break whatever poorly understood mechanism has been responsible for humans making moral/philosophical progress in the past.
Whoever does take over the world, if they were sufficiently wise (i.e., concerned about their own moral progress), could deliberately institute new mechanisms to support moral progress with AI help. I think historically, decentralization of power and competition has (apparently) been better for moral progress, but this may not hold in the future.
And third, you don’t get the potential huge benefits from trade. I’ve argued that it’s quite unlikely that most people will converge on the best moral views. So, while there’s some small chance that a dictator ends up having good moral views and acting on them, it’s probably not the case. And then there’s no one else to trade with.
This seems to ignore acausal trade (which I see you’ve acknowledged as a possibility elsewhere). Compared to the kind of trade you talk about, I think I’m more worried that in a more decentralized/competitive world, philosophical progress might get derailed more easily (due to above concerns), causing acausal trade or other opportunities that are potentially vastly more beneficial (see Beyond Astronomical Waste) to be lost.
Why are AI-enabled coups worse than the alternative in expectation? I’m personally quite uncertain about this. I looked around a bit and found an explanation from you in this podcast, which I quote below and respond to. Please let me know if you have a more detailed writeup somewhere.
I agree this has historically been the case, but think the selection pressure could be a bit different with AI-enabled global takeover, which perhaps require somewhat less political maneuvering and will to dominate other humans, and more technical foresight/understanding and appreciation of astronomical stakes. It seems unclear to me that on expectation the kind of person who becomes an AI-enabled global dictator would have worse values than average. (Of course “worse than ideal” is much more likely, but seems like the wrong standard to use here? Morality is Scary gives some explanations for my relatively low opinion of average human’s values.)
I’m afraid that such mechanisms would probably go away even without intense concentration of power due to considerations like these. In short, AI-optimized persuasion would likely replace human debate and break whatever poorly understood mechanism has been responsible for humans making moral/philosophical progress in the past.
Whoever does take over the world, if they were sufficiently wise (i.e., concerned about their own moral progress), could deliberately institute new mechanisms to support moral progress with AI help. I think historically, decentralization of power and competition has (apparently) been better for moral progress, but this may not hold in the future.
This seems to ignore acausal trade (which I see you’ve acknowledged as a possibility elsewhere). Compared to the kind of trade you talk about, I think I’m more worried that in a more decentralized/competitive world, philosophical progress might get derailed more easily (due to above concerns), causing acausal trade or other opportunities that are potentially vastly more beneficial (see Beyond Astronomical Waste) to be lost.