Founder and CEO of ControlAI.
Andrea_Miotti
Karma: 1,947
Anthropic did not call for a pause on AI
ControlAI leadership has been quite consistently deceptive and engaged in strategic conflation of AI existential risk
Not only have we been talking about x-risk consistently for years, but we were also essentially the first ones to systematically talk about it in mainstream media. We were also among the first to push people in the AI safety sphere to be more vocal about it.
For example, in 2023, I wrote an op-ed in TIME, where I argued that strong international coordination is needed to prevent x-risk from AI. Connor has one of the first major TV appearances where anyone talks about x-risk as something that can come from scaling the current paradigm, back in 2023.
Here’s just a few examples of us publicly speaking about x-risk in the last few years alone:
(September 2023) Connor Leahy addresses the UK House of Lords about extinction risk from AGI.
(October 2023) Connor Leahy at the Cambridge Union debate “This House Believes Artificial Intelligence Is An Existential Threat.”
(October 2024) Connor Leahy, Gabriel Alfour, and Andrea Miotti coauthor The Compendium, a guide laying out why AGI/ASI poses an extinction risk.
(October 2024) Andrea Miotti publishes A Narrow Path, a policy plan on how humanity can survive and prevent ASI development.
(October 2025) Andrea Miotti op-ed in TIME “We Need a Global Movement to Prohibit Superintelligent AI.”
(December 2025) Connor Leahy and Gabriel Alfour testify about x-risk from ASI to the Canadian House of Commons Ethics Committee.
(March 2026) Andrea Miotti testifies about x-risk from ASI to the Canadian House of Commons Ethics Committee.
(April 2026) Connor Leahy testifies about x-risk to the Canadian Senate Standing Committee on Transport and Communications.
(May 2026) Connor Leahy speaks on extinction risk from ASI on Al Jazeera “Will AI rule the world?”
(May 2026) Andrea Miotti on BBC News “Andrea Miotti Explains the Superintelligence Threat.”
(May 2026) Andrea Miotti op-ed in The Spectator “We are closer to AI extinction than we think”
If we win: first order of business is to celebrate and give the team a long holiday! The most likely cause of death of the authors has been averted.
After that, the work is building the institutions and societal infrastructure so humanity can survive and thrive alongside very powerful AI (and increasingly powerful technology in general). Even with a global ASI ban, powerful AI systems will still exist and society will still go through radical disruption. All versions of the future are wild, even conditional on a ban.
And a ban isn’t a one-and-done. Humanity will be protected from self-annihilation only if there is continued vigilance, and the institutions and people that enforce the ASI regime are maintained and updated as technological progress continues.ControlAI winding down after winning would be a completely fine outcome. Whether phase two I describe above is ControlAI’s focus or a new venture’s is a good question for after we’ve prevented extinction risk from ASI.
Whether that work eventually entails finding ways to safely develop ASI under operationally adequate institutions and just processes, or not, is unclear to us. And in any case, a globally enforced ASI ban is the precondition for ever doing a “safe ASI” plan.
Personally, there are many other causes I care about, such as extending human lifespan and updating political institutions to deal with 21st century technology, that will become higher priority once the largest threat to the continued existence of humanity is averted.
The political coalition that ends up banning ASI includes a generalised anti-technology faction which continues to gain in power and attempt to ban more technologies. Is your org working against this, or is it somebody else’s job.
You’re right that this could happen, and we’d consider it a bad outcome, although a better one than extinction. We don’t support a universally anti-technology coalition, which is one of the many reasons we take care to keep our messaging focused specifically on ASI risk rather than omnicause anti-technologism.
Nice post, strong upvoted. You might be interested in a related post also dealing with the illusion of “escaping the permanent underclass”: https://cognition.cafe/p/the-realpolitik-of-the-permanent
Yup, he’s already there! Connor moved to America recently and he’s based in Washington, DC, leading our work in the US!
See: https://www.ettf.land/p/conjecture-a-retrospective
Thanks Michael!
I expect the third, fourth, fifth etc. person we hire to be at least as effective at getting those meetings as current staff were at equivalent tenure, and probably more so. I also expect the team in aggregate to get more effective over time: this is what has happened over and over for each of our workstreams.
This is for a few reasons, also listed in the post:
-
ControlAI’s success relies on strong processes and infrastructure, not on pre-existing political insider networks or rare and obscure talent. Our lawmaker efforts all started with staff members (who are great!) with no insider networks, and between 0 and a few years of experience in policy. We don’t succeed based on insider connections: we succeed because of our direct, scalable method. New hires also inherit advantages the first hire didn’t have: a refined playbook and existing campaign traction to point to.
-
Success compounds vertically in each country, and horizontally across countries. a. Vertically, within a single country, the more lawmakers support a campaign as a result of our meetings, the more other lawmakers learn about extinction risk from AI and the need to tackle the threat from superintelligent AI, making it marginally easier to get more meetings and more lawmakers on board. This is a micro-example of our macro-strategy: the bottleneck right now is lack of awareness, and building common knowledge leads to faster and faster change. b. Horizontally, across countries, the more lawmakers in a major country support the campaign, the (slightly) easier it is for lawmakers in another country to take the topic seriously, meet us, and support. In the UK, we started from 0. In Canada, Germany and the US, we started with “Dozens of UK lawmakers, across parties, recognize extinction risk from AI and superintelligence as a national security threat”.
-
Our approach compounds across ControlAI as a whole. Every week, our briefing gets better: our materials, processes, arguments and pitch are improved continuously based on real-world feedback from the meetings we have. Each iteration is propagated across the org.
Given this, the main scaling risk here for me is not that hire 3 underperforms hires 1 and 2. It’s the normal risk of hiring and onboarding well, which we’re actively investing in right now with more onboarding materials, internal and external writeups of our theory of change and our approach, and even more streamlined processes.
-
My estimate accounts for increased, adversarial lobbying by frontier AI corporations regardless as they get closer and closer to superintelligence.
I do not expect us being heavily funded to change that by a very significant degree.
We can already see right now that AI corporations are ramping up their lobbying and influence operations immensely, across the board. To name just a few examples:
DC lobbyists received nearly $130M for work related to AI in 2025, an increase of 370% since the release of ChatGPT https://news.bgov.com/bloomberg-government-news/ai-influence-spending-booms-signaling-monumental-clashes-ahead
Anti AI-regulation SuperPAC coordinated by Chris Lehane (chief OpenAI lobbyist), funded by OpenAI’s Greg Brockman and A16Z, raising more than $125M to spend on electoral races against pro-AI regulation candidates. https://www.cnbc.com/2026/01/30/ai-industry-super-pac-raises-campaign-money.html
Anthropic ramping up its own SuperPAC contributions. https://builtin.com/articles/super-pacs-ai-regulation-2026-midterms
It’s a little hard for me to understand whether you agree or disagree with the plan in the post for me.
My reading of the piece you posted is that it was correctly identifying (23 years ago!) the importance of the political process in avoiding bad outcomes from AI, and was particularly prescient.
To highlight the key part that I’d like people to take away, “only a strong public movement driving government regulation” has a chance of solving the problem.
I completely agree, and our plan at ControlAI and the post above explains how our plan focuses on informing both the public and lawmakers, at scale, about the extinction risk posed by superintelligence and the way to prevent these risks.
Preventing extinction from ASI on a $50M yearly budget
Thanks Charlotte! Yes, a big part of our approach is based on building common knowledge of the risk of extinction from superintelligence, so people know that other people know about this risk.
The more people we reach, the faster things can compound.
ControlAI 2025 Impact Report: our progress toward an international ban on ASI
How middle powers may prevent the development of artificial superintelligence
Modeling the geopolitics of AI development
Three main views on the future of AI
Anthropic CEO calls for RSI
The Compendium, A full argument about extinction risk from AGI
Thanks! Do you still think the “No AIs improving other AIs” criterion is too onerous after reading the policy enforcing it in Phase 0?
In that policy, we developed the definition of “found systems” to have this measure only apply to AI systems found via mathematical optimization, rather than AIs (or any other code) written by humans.
This reduces the cost of the policy significantly, as it applies only to a very small subset of all AI activities, and leaves most innocuous software untouched.
A Narrow Path: a plan to deal with AI extinction risk
In terms of explicit claims:
“So one extreme side of the spectrum is build things as fast as possible, release things as much as possible, maximize technological progress [...].
The other extreme position, which I also have some sympathy for, despite it being the absolutely opposite position, is you know, Oh my god this stuff is really scary.
The most extreme version of it was, you know, we should just pause, we should just stop, we should just stop building the technology for, indefinitely, or for some specified period of time. [...] And you know, that extreme position doesn’t make much sense to me either.”
Dario Amodei, Anthropic CEO, explaining his company’s “Responsible Scaling Policy” on the Logan Bartlett Podcast on Oct 6, 2023.
Starts at around 49:40.
- AI #35: Responsible Scaling Policies by (26 Oct 2023 13:30 UTC; 66 points)
- It will cost you nothing to “bribe” a Utilitarian by (15 Oct 2025 15:51 UTC; 44 points)
- Lying is Cowardice, not Strategy by (24 Oct 2023 13:24 UTC; 19 points)
- Lying is Cowardice, not Strategy by (EA Forum; 25 Oct 2023 5:59 UTC; -5 points)
Please share what “bad things” you are accusing us of then, with actual sources.
Habryka has made vague accusations against ControlAI for years, and his comments in this thread look like just the latest iteration of this.
Being consistently public about extinction risks, having it on our homepage, and having 130+ lawmakers publicly supporting our campaigns that explicitly mentions AI x-risk makes it much harder to “conflate them”.
Happy to engage with the actual points if you have links to things we’ve done you think are bad.