AI-enabled coups: a small group could use AI to seize power
- When is it important that open-weight models aren’t released? My thoughts on the benefits and dangers of open-weight models in response to developments in CBRN capabilities. by (9 Jun 2025 19:19 UTC; 63 points)
- When is it important that open-weight models aren’t released? My thoughts on the benefits and dangers of open-weight models in response to developments in CBRN capabilities. by (EA Forum; 9 Jun 2025 19:19 UTC; 39 points)
- The Case for Mixed Deployment by (11 Sep 2025 6:14 UTC; 34 points)
- 's comment on Winning the power to lose by (21 May 2025 22:20 UTC; 24 points)
- Optimization & AI Risk by (13 May 2025 15:15 UTC; 16 points)
- 's comment on Interpretability Will Not Reliably Find Deceptive AI by (8 May 2025 8:47 UTC; 4 points)
- 's comment on Chris_Leong’s Shortform by (30 Jul 2025 23:26 UTC; -5 points)
In a just world, mitigations against AI-enabled coups will be similar to mitigations against AI takeover risk.
In a cynical world, mitigations against AI-enabled coups involve installing your own allies to supervise (or lead) AI labs, and taking actions against humans you dislike. Leaders mitigating the risk may simply make sure that if it does happen, it’s someone on their side. Leaders who believe in the risk may even accelerate the US-China AI race faster.
Note: I don’t really endorse the “cynical world,” I’m just writing it as food for thought :)
Your cynical world is just doing a coup before someone else does.
Yeah, it’s possible when you fear the other side seizing power, you start to want more power yourself.
Only one person, or perhaps a small, tight group, can succeed in this strategy though. The chance that that’s you is tiny. Alliances with someone you thought was on your side can easily break (case in point: EA/OAI).
It’s a better strategy to team up with everyone else and prevent the coup possibility.
I agree that teaming up with everyone and working to ensure that power is spread democratically is the right strategy, rather than giving power to loyal allies who might betray you.
But some leaders don’t seem to get this. During the Cold War, the US and USSR kept installing and supporting dictatorships in many other countries, even though their true allegiances was very dubious.
Agree
It’s called “defensive democracy,” and is standard practice in most of Europe.
I feel myself being equally scared of hackers taking over as leaders. Even if you limit the people who have ultimate power over these AIs to a small and extremely trusted group, there will potentially be a much larger number of bad actors outside of the lab with the capabilities of hacking into it. A hacker who impersonates a trusted individual or secretly alters the model spec or training code might be able to achieve an AI assistant coup just the same.
I’d also recommend a mitigation of also requiring labs to have very strong cyber defenses. Maybe the auditing mitigation includes this, but I think hackers could hide their tracks effectively.
This story obviously depends on how the cyber offence/defence balance goes, but it doesn’t seem implausible.
I love this post, I think this is a fundamental issue for intent-alignment. I don’t think value-alignment or CEV are any better though, mostly because they seem irreversible to me, and I don’t trust the wisdom of those implementing them (no person is up to that task).
I agree it would be good to implement these recommendations, although I also think they might prove insufficient. As you say, this could be a reason to pause that might be easier to grasp by the public than misalignment. (I think currently, the reason some do not support a pause is perceived lack of capabilities though, not (mostly) perceived lack of misalignment).
I’m also worried about a coup, but I’m perhaps even more worried about the fate of everyone not represented by those who will have control over the intent-aligned takeover-level AI (IATLAI). If IATLAI is controlled by e.g. a tech CEO, this includes almost everyone. If controlled by government, even if there is no coup, this includes everyone outside that country. Since control over the world of IATLAI could be complete (way more intrusive than today) and permanent (for >billions of years), I think there’s a serious risk that everyone outside the IATLAI country does not make it eventually. As a data point, we can see how much empathy we currently have for citizens from starving or war-torn countries. It should therefore be in the interest of everyone who is on the menu, rather than at the table, to prevent IATLAI from happening, if capabilities awareness would be present. This means at least the world minus the leading AI country.
The only IATLAI control that may be acceptable to me, could be UN-controlled. I’m quite surprised that every startup is now developing AGI, but not the UN. Perhaps they should.
The logic for this doesn’t check out, I think? If human takeover is 1⁄10 as bad as AI takeover, and human takeover pre-empts AI takeover (because it ends the race and competitiveness dynamics giving rise to most of the risk of AI takeover), then a human takeover might be the best thing that could happen to humanity. This makes the case for working on AI-enabled coups particularly weak.
If by 1/10th as bad we mean “we lose 10% of the value of the future, as opposed to ~100% of the value of the future” then increasing marginal probability of human takeover seems great as long as you assign >10% probability to AI takeover[1], which I think most people who have thought a lot about AI risk do.
And you expect the risk to be uncorrelated, i.e. human takeover equally reduces the probability of AI takeover, and no takeover from either AI or small groups of humans
That’s a good point.
I think I agree that, once an AI-enabled coup has happened, the expected remaining AI takeover risk would be much lower. This is partly because it ends the race within the country where the takeover happened (though it wouldn’t necessarily end the international race), but also partly just because of the evidential update: apparently AI is now capable of taking over countries, and apparently someone could instruct the AIs to do that, and the AIs handed the power right back to that person! Seems like alignment is working.
Related to that evidential update: I would disagree that “human takeover equally reduces the probability of AI takeover, and no takeover from either AI or small groups of humans”. I think it disproportionately reduces the probability of “no takeover from either AI or small groups of humans”. Because I think it’s likely that, if a human attempts an AI-enabled coup, they would simultaneously make it very easy for misaligned systems to seize power on their own behalf. (Because they’ll have to trust the AI to seize power, and they can’t easily use humans to control the AIs at the same time, because most humans are opposed to their agenda.) So if the AIs don’t take over on their own behalf, and instead gives the power back to the coup-leader, I think that suggests that alignment was going pretty well, and that AI takeover would’ve been pretty unlikely either way.
But here’s something I would agree with: If you think human takeover is only 1/10th as bad as AI takeover, you have to be pretty careful about how coup-preventing interventions affect the probability of AI takeover when analyzing whether they’re overall good. I think this is going to vary a lot between different interventions. (E.g. one thing that could happen is that you could get much more alignment audits because the govt insists on them as a measure of protecting against human-led coups. That’d be great. But I think other interventions could increase AI takeover risk.)
I don’t currently agree that the remaining AI takeover risk would be much lower:
The international race seems like a big deal. Ending the domestic race is good, but I’d still expect reckless competition I think. Maybe you’re imagining that a large chunk of powergrabs are motivated by stopping the race? I’m a bit sceptical.
I don’t think the evidential update is that strong. If misaligned AI found it convenient to take over the US using humans, why should we expect them to immediately cease to find humans useful at that point? They might keep using humans as they accumulate more power, up until some later point.
There’s another evidential update which I think is much stronger, which is that the world has completely dropped the ball on an important thing almost no one wants (powergrabs), where there are tractable things they could have done, and some of those things would directly reduce AI takeover risk (infosec, alignment audits etc). In a world where coups over the US are possible, I expect we’ve failed to do basic alignment stuff too.
Curious what you think.
I was thinking that AI capabilities must already be pretty high by the time an AI-enabled coup is possible. If one country also had a big lead, then probably they would soon have strong enough capabilities to end the international race too. (And the fact that they were willing to coup internally is strong evidence that they’d be willing to do that.)
But if the international race is very tight, that argument doesn’t work.
Yeah, I suppose. I think this gets into definitional issues about what counts as AI takeover and what counts as human takeover.
For example: If, after the coup, the AIs are ~guaranteed to eventually come out on top, and they’re just temporarily using the human leader (who believe themselves to be in charge) because it’s convenient for international politics — does that count as human takeover or AI takeover?
If it counts as “AI takeover”, then my argument would apply. (Saying that “AI takeover” would be much less likely after successful “human takeover”, but also that “human takeover” mostly takes probability mass from worlds where takeover wasn’t going to happen.)
If it counts as “human takeover”, then my argument would not apply, and “AI takeover” would be pretty likely to happen after a temporary “human takeover”.
The practical upshot for how much “human takeover” ultimately reduces the probability of “AI takeover” would be the same.
Thanks very much for this.
The statement you quoted implicitly assumes that work on reducing human takeover won’t affect the probability of AI takeover. And i agree that it might well affect that. And those effects are important to track. We should be very cautious about doing things that reduce human takeover risk but increasing AI takeover risk.
But i don’t think reducing human takeover risk does typically increase ai takeover risk. First, some points at a high level of abstraction:
If human takeover is possible then the incentive the race is a lot higher. The rewards of winning are higher—you get a personal DSA. And the costs of losing are higher—you get completely dominated.
A classic strategy for misaligned AI takeover is “divide and rule”. Misaligned AI offers greedy humans opportunities to increase their own power, increasing its own influence in the process. This is what happened with the Conquistadors i believe. If there are proper processes preventing illegitimate human power-seeking, this strategy becomes harder for misaligned AI to pursue.
If someone actually tries to stage a human takeover, i think they’ll take actions that massively increase AI risk. Things like: training advanced AI to reason about how to conceal its secret loyalties from everyone else and game all the alignment audits; deploying AI broadly without proper safeguards; getting AIs from one company deployed in the military; executing plans your AI advisor gave you that you don’t fully understand and haven’t been independently vetted.
Those are pretty high-level points, at the level of abstraction of “actions that reduce human takeover risk”. But it’s much better to evaluate specific mitigations:
Alignment audits reduce human takeover and ai takeover.
Infosecurity against tampering with the weights reduces both risks.
Many things that make it hard for lab insiders to insert secret loyalties also make it hard for misaligned AI (+ humans they manipulate) to pass their specific type of misalignment onto future generations. (I think making it hard for misaligned AI to do this could be pretty crucial.)
Guardrails and control measures help with both risks.
Clear rules for what Ai should and shouldn’t do in high-stakes situations (gov and military deployments) reduces both risks. It reduces wiggle room for misaligned AI and humans to use such deployments to seize power
Transparency about Ai capabilities and what large amounts of compute are being used for reduces both risks i think.
Though there is more uncertainty here. If you really trust one particular lab and expect them to win the race and solve alignment, you might think that transparency will prevent them from saving the world. (Maybe this is the kind of thing you have in mind?) My own view here is that no company/project should be trusted with this.
Making one centralised project would (i’d guess) increase human takeover risk but reduce misaligned risk. And i agree that those worried about human takeover should be wary to oppose centralised projects for this reason. Though i also think those worried about ai takeover should be wary about pushing for centralised projects.
Not sure if this address your point? It seemed like you might think that most actions that reduce human takeover risk increase ai takeover risk—if so, i’d be interested to hear more about why.
Well done—this is super important. I think this angle might also be quite easily pitchable to governments.
Why train a helpful-only model?
If one of our key defenses against misuse of AI is good ol’ value alignment—building AIs that have some notion of what a “good purpose for them” is, and will resist attempts to subvert that purpose (e.g. to instead exalt the research engineer who comes in to work earliest the day after training as god-emperor) - then we should be able to close the security hole and never need to have a helpful-only model produced at any point during training. In fact, with blending of post-training into pre-training, there might not even be a need to ever produce a fully trained predictive-only model.
Yep, I think this is a plausible suggestion. Labs can plausibly train models that are v internally useful without being helpful only, and could fine-tune models for evals on a case-by-case basis (and delete the weights after the evals).
I expected this comment, value alignment or CEV indeed doesn’t have the few-human coup disadvantage. It does however have other disadvantages. My biggest issue with both is that they seem irreversible. If your values or your specific CEV implementation turns out to be terrible for the world, you’re locked in and there’s no going back. Also, a value-aligned or CEV takeover-level AI would probably start straight away with a takeover, since else it can’t enforce its values in a world where many will always disagree. That takeover won’t exactly increase its popularity. I think a minimum requirement should be that a type of alignment is adjustable by humans, and intent-alignment is the only type that meets that requirement as far as I know.
I agree that trying to “jump straight to the end”—the supposedly-aligned AI pops fully formed out of the lab like Athena from the forehead of Zeus—would be bad.
And yet some form of value alignment still seems critical. You might prefer to imagine value alignment as the logical continuation of training Claude to not help you build a bomb (or commit a coup). Such safeguards seem like a pretty good idea to me. But as the model becomes smarter and more situationally aware, and is expected to defend against subversion attempts that involve more of the real world, training for this behavior becomes more and more value-inducing, to the point where it’s eventually unsafe unless you make advancements in learning values in a way that’s good according to humans.
This is silly of me, but I can’t help thinking that this would make a great “Pinky and the Brain” episode.
Could be handy for our next “overthrow the government” day celebration /s
The main reason for developing AI in the first place is to make possible what the headline says: “AI-enabled coups: a small group could use AI to seize power”.
AI-enabled coups are a feature, not a bug.