Michaël Trazzi
pause some frontier RL training
How should we update on OpenAI’s Astra coming up with 10 new math and CS results?
How uplifted are mathematicians by Astra? How much is this uplift accelerating / a bottleneck to the automated AI R&D loop? How different would it be if it was AI coming up with completely new math & frameworks vs. closing the gaps from current litterature? @Daniel Kokotajlo @elifland
For instance, regarding the non sofic group, Fable was initially saying it was the most impressive of these 10 results. But also, it’s apparenlty based on some Kun-Thom paper from 2026 and a framework from 2019 from Kun-Thom too, so somehow filling the gaps from recent papers? (Note: I have looked a very limited time into this, so probably missing a lot of information here).
“no CEOs have signed it”
Website says that Dario signed it.
Also, protest at OpenAI. I know people who want to organize something. DM me on Signal for details. (mtrazzi.99)
Last time we protested xAI, but decided to not protest them this time. They’d be included in major AI companies.
Chinese labs would stop if there was a bilateral agreement between the US and China.
In four days (July 11, 12-4pm), about ~200 of us (event link), including 13 different groups, will be marching on OpenAI, Anthropic and Google in SF asking the CEOs to commit to stop developing more powerful models if every other major AI company (and China) does the same. We’re calling this The AI Protest (theprotest.ai).
We’re gonna have some of the speakers from last time (Nate Soares (MIRI), David Krueger (Evitable), Will Fithian (Berkeley Professor)) but also try to get folks from different groups part of the coalition speaking too.
As Scott Alexander puts it: “Participants are about half from our conspiracy and half from random anti-data-center-type groups, which I think is how this basically has to work, so don’t be surprised if you run into the latter.”
On why we’re doing this, see:The blurb in the stoptherace.ai website where I explain that “expects” from CEOs isn’t actually committing, and why we need committments
Ronak’s post on why you should come to the protest
My theory of change answering Katja’s Pause post
Transformers’ The AI Safety Movement needs normies
I would add:
Marching together is fun. Here’s the data from a survey after the protest (n=25 respondents, about 12.5% of the protesters).
Some of the risks described in this post can be mitigated (eg. by wearing a mask).
Great post! Agree with all of the above.
For folks who want to pursue the Pause ASAP case, about 110+ of us will be marching on OpenAI, Anthropic and Google on July 11, asking Sam Altman, Dario Amodei and Demis Hassabis to make public commitments about pausing frontier AI development (if others also pause).
Sign up here: https://luma.com/s0k8wvee
Or here: https://partiful.com/e/EChgbBMsoeN3qVnATHKJ
---
How I think protesting now helps us get to an AI Pause / International Treaty ASAP:
Coalition Building with other organisations that are already activated on other issues, and are starting to get worried about AI. Our July 11 protest will be part of a larger coalition (theprotest.ai). Right now it’s featuring 6 orgs, mostly AI Safety, but including The National Union of Healthcare Workers. We expect more orgs like the league of women voters to join next week.
Raising political salience through media attention. Last march was featured in major newspapers in the US (New York Times (1, 2), Washington Post, The Atlantic), MS NOW (prev MSNBC), and major news outlets in Brazil / Spain.
Pressuring the labs to making more conditional statements about pausing. After our protest, Anthropic wrote it expects it “would slow down or temporarily pause” if other labs verifiably did too, and OpenAI wrote it expects coordination, including slowing frontier development, to become more important. We have reasons (that we can only keep private for now) on why we believe our protest was instrumental for OpenAI, in particular for their first blogpost.
Putting an AI Pause at the center of the public discourse, so that AI Lab CEOs can’t escape it. This was what happened with the Google DeepMind Hunger Strike, where months later a journalist (Emily Chang) asked Demis Hassabis the same question we were asking him, which lead to him making a statement (or at least a comment / answer) on pausing AI if everyone else pauses.
The process of making your own complicated model like this, and engaging with the models made by others, is… well, I think it’s pretty edifying.
I second this.
I’d add:If you disagree with someone’s predictions / timelines, try to figure out why.
I remember disagreeing with someone with experience in AI forecasting about their longer timelines, and trying to figure out exactly what parameters we disagreed on on https://www.aifuturesmodel.com/ was a useful exercise.
More generally, try coming up with variables from models from scratch and then look at why your guess differ from other people’s guess
Try to explain the model to other people. Could be in writing, through videos, your friends, etc.
Try to understand the limitations of other people’s models, and potentially come up with better ones (like is Automated Coder really a useful threshold? Is research taste that important?)
Regarding the CNN interview
You say:
Asked about the downside, Clark sidestepped that very loss-of-control scenario and moved to talking about how we could verify and trust these systems, comparing it to dropping “hundreds or thousands of new colleagues” into the newsroom.
But the actual exchange on the downsides was (emphasis mine):
Cooper: The downside — for anybody who’s seen any science fiction movie… in all the science fiction movies, we give control to these machines, and we all know what happens — the people who create them are the first ones who get killed. What to you is the risk here?
Clark: We read the science fiction and watch science fiction here as well, so it’s not lost on us — this is how some of the stories start. The risk here is what happens if you can’t validate or verify or trust the behavior of these systems. It would be like if we dropped hundreds or thousands of new colleagues into your newsroom — it would take you a while to figure out if you can trust them, if they work the way you expect, if when you ask them to do things they come back with something good and in line with your expectations. That’s one of the challenges: how do you maintain control over fleets of scientists that are much, much larger and much faster than ones you’ve had before?
So Clark reframed the question in terms of verification and trust, but ended his answer on loss of control.
Similarly, you write:
And when Anderson Cooper asks whether Anthropic wants to see the industry as a whole slow or pause AI development, Clark’s answer opened bluntly: “Our view is we’ve built amazingly powerful technology. We’re going to keep building it.”
But here you’re not quoting the rest of his answer where he’s actually talking about removing our foot from the gas pedal and stop accelerating (emphasis mine):
Clark: Our view is we’ve built amazingly powerful technology, we’re going to keep building it, and in coming years that technology is going to start to do a lot of major things in the world in domains like science. But when I look down at the car we’re driving, all I have is a gas pedal — I don’t have a brake pedal. Surely at some point in the future we might want that option: to say to ourselves, to other companies, to the world — what would it be like if we focus now on taking these scientific advances we’ve created and pushing them through to the world, and take our foot off the gas of just accelerating the AI systems?
So while I agree that Dario has not been calling for a pause, it seems to me that Jack Clark’s post on RSI is more coherent with what he said in his CNN interview than your post suggests.
Regarding Anthropic Calling for a Pause
You also write that it’s part of a pattern of making “no concrete commitments”:
It’s a deliberate PR approach that Anthropic and OpenAI have used over and over to curry favor with multiple opposing audiences, while making no concrete commitments.
And that AI companies will not lead a slowdown:
Anthropic’s blog post’s language is deliberately vague, underscoring that companies will not lead a slowdown.
But Jack Clark ends the RSI blogpost with an explicit commitment to organize conversations on RSI and coordination, and to publish the results:
In the coming months, we will organize conversations where policymakers, researchers, civil society, and other AI companies can help answer some of the questions this piece raises, especially around full recursive self-improvement and how to create better options for coordination and deliberation. We’ll publish what comes out of it. The window to investigate the questions together is here, and people outside AI companies should be involved in this deliberation.
I agree that it’s a much weaker commitment than the actual pause commitments we need. And obviously “organizing conversations” about RSI & “better options for coordination and deliberation” is definitely not the same as advocating for a pause.
But I do think Jack Clark is taking some steps here to open up the conversation on RSI & pausing.
I think that Joseph Miller is one of the most competent people in the space and I think PauseAI UK should continue to get funded.
However, there are some things that I think would be worth clarifying in your post:
On the Political work
We wrote a memo which was sent to all MPs prior to the debate and drafted some of the speeches, putting us in a strong position to work with those MPs when proposing amendments to the Cyber Security and Resilience Bill.
How do you know it put you in a stronger position? Like how many MPs do you know used some of your speech notes, and how many strong relationships did you develop?
In October we held a screening in the UK Parliament of filmmaker Michaël Trazzi’s documentary about SB-1047, the proposed California AI legislation. This helped to inform MPs and Peers about the kinds of AI legislation that could be in a UK AI bill
Thanks again for hosting this! Can you clarify for readers how many MPs actually came to the movie screening & talked to you guys? I think it makes a difference to know if it was like 1-2 MPs or dozens. (Note: I remember you telling me it was more like the former, happy for you to rectify).
On The Hypothetical Scenarios
Now, regarding the hypothetical scenarios:
PauseAI UK has 10,000 highly dedicated volunteers who act as a dominant lobbying force on AI policy matters.
How realistic is this scenario? Like how many dedicated volunteers does PauseAI currently have? Without mentioning the timeline and the likelihood of this scenario, I’m wondering if you’re thinking of a 0.1% ideal scenario over 5 years, or if you’re actually 50% confident this could happen in 2-3 years. Like if you currently have 5-10 dedicated volunteers spending several hours (> 5 hours) a week working on PauseAI UK, then 10,000 would be more than 10 doublings, so ~6 years if we believe the number of dedicated volunteers follow the same exponential growth as you claim for protests?
Now regarding the other scenario:
PauseAI protests double in size every 7 months as AI capability itself improves exponentially. [...] PauseAI UK organises a march in Westminster with 1 million attendees and dominates headlines in the British press. The prime minister is obliged to respond and commits to opening negotiations for a global pause agreement.
OK so basically, PauseAI UK continue doing protests, the 300 people early 2026 protest becomes 600 by October 2026, and so forth and so on until… Feb 2029 when we get TED-AI (if we believe the AI Futures project people)? In that case, we’d have ~4 more doublings, aka ~10k people protesting, so the 1M number is actually a 100x of the default case without warning shot.
To estimate the expected value of that scenario, and how useful having PauseAI UK’s impact was there, we should consider:
How likely (P_dou) is that trend of doubling every 7 months to hold in the future?
How likely (P_war) is it that we get a specific warning shot big enough to give you a big enough increase (eg. the 100x mentioned before) in protest size?
How likely (P_gov) is it that your government will actually do something based on the protest?
So very roughly, that scenario would be about like P_dou x P_war x P_gov likely. In my view, if I was to give very rough numbers that’d be about 0.5 x 0.1 x 0.1 = 0.5% likely.
And then you’d need to also factor in how likely would it be for a protest of this size to happen without PauseAI, and how much leverage the UK would have to lead us to an actual treaty anyway.
Regarding the Exponentials and 7 months doubling
You probably have way more context on this than me, but from looking at the data quickly my reading is:
You basically had the same turnout from Nov 2023 and May 2024
Then on Feb 2025 there was a surprisingly low datapoint
Then on Jun 2025 you had this protest in front of DeepMind where 100+ people signed up but you got closer to 75 in turnout?
Then on Feb 2026 you organize this event that got 227 signups on luma, and ~300 people showed up
Some comments on this:
For the last point, you mix Pull the Plug and PauseAI in the number of people who came, but you don’t mix them in the signups. So what seems like more people showing up than signing up is actually because Pull the Plug had a different sign up page.
From what you say there was the same amount of people who came from Pull the Plug and PauseAI. So about 150.
So essentially, per the metric of how many people actively asking for a Pause, it seems like we were closer to 150 by Feb 2026. So the number of attendes specifically for pausing were like 20, 20, 40, 150 for these 4 datapoints over 2.5 years, which is not clearly a 7-months doubling / exponential?
Again, I don’t want to say that your work is not valuable. I think AI Safety activism is probably one of the most neglected things to do. And I really hope you get funded.
But I think there are many points in this post that would be worth clarifying.
[Note: wrote this quickly, might include errors]
.
I do agree that “asserts” was too strong. Changed to “writes about the possibility of needing to coordinate with governments and other labs before proceeding further” to stay closer to the quote.
That said, I still think interpreting the “before” part as a statement about pausing (even for a short time) is a reasonable interpretation.
In a blogpost posted yesterday, Sam Altman writes about the possibility of needing to coordinate with governments and other labs before proceeding further (emphasis mine):
We expect there will be periods where we need to collaborate with governments, international agencies, and other AGI efforts to ensure that we have sufficiently solved serious alignment, safety, or societal problems before proceeding further with our work.
This comes one month after Stop The AI Race’s March 21st protest in front of OpenAI, Anthropic and xAI (which I organized), asking Sam Altman (alongside other CEOs) to make a statement on pausing frontier AI development (conditionally), and a follow-up direct message on March 25th asking Sam Altman to clarify his take on conditionally pausing AI. (The Musk v. Altman trial also begins today, which may be relevant to the timing.)
Other parts of the blogpost also point towards more coordination with other labs and governments:
“we need to ensure that key decisions about AI are made via democratic processes and with egalitarian principles, and not just made by AI labs.”
″AI will introduce new risks, and we will work with other companies, ecosystems, governments, and society to solve them. “
”No AI lab can ensure a good future alone. For an obvious example, there may be extremely capable models that make it easier to create a new pathogen, and we need a society-wide approach to defend against this with pathogen-agnostic countermeasures.”
I’ve been pretty impressed with ControlAI’s team & ability to talk to many policymakers in the UK & US overall.
At the moment, we are cautiously optimistic: in the past 5 months, with ~1 staff member,[16]we’ve managed to personally meet with and brief 18 members of Congress, as well as over 90 Congressional offices.
Footnote 16 says:
1 member for most of this period; the 2nd member joined in the past month.
How successful do you expect the third, fourth, fifth, etc. person you hire to be at getting those meetings?
Yeah, some people who have been flyering for this have noticed that most people just take a picture of the flyer & don’t bother to actually RSVP to the protest (sometimes for privacy reasons). We’ll see how many people end up coming!
More people show up on weekends yeah
Removed the quietly and linked to Holden’s post, thanks!
In two days (March 21st, 12-4pm), about 140 of us (event link) will be marching on Anthropic, OpenAI and xAI in SF asking the CEOs to make statements on whether they would stop developing new frontier models if every other major lab in the world credibly does the same. This comes after Anthropic removed its commitment to pause development from their RSP.
We’ll be starting at 500 Howard St, San Francisco (Anthropic’s Office, full schedule and more info here). This is shaping to be the biggest US AI Safety protest to date, with a coalition including Nate Soares (MIRI), David Krueger (Evitable), Will Fithian (Berkeley Professor) and folks representing PauseAI, QuitGPT, Humans First.
tl;dr: Stop The AI Race is organizing some rapid-response protest march this Thursday 9am-12pm in San Francisco, calling the Mayor of San Francisco and the board of supervisors to declare an AI Emergency and enforce a pause of frontier AI model development in San Francisco
Over the past week, we have seen Dario Amodei and Sam Altman advocate for government intervention in pacing the frontier.
However, the US Government has yet to answer their calls. Which is why on Thursday at 9am Stop The AI Race will be organizing another protest march, asking local authorities to step in.
We will be marching from OpenAI to Anthropic to City Hall, asking AI company employees to join us in calling on the Mayor of San Francisco and the Board of Supervisors to declare an AI Emergency and enforce a full pacing of frontier AI model development in San Francisco.
Schedule:
- 9am: Rally at OpenAI
- 10-10:45am: March To Anthropic
- 10:45am: Rally at Anthropic
- 11am-11:30am: March to City Hall
- 11:30am: Rally at City Hall
Sign up here.