CEO of PauseAI. Ex AI engineer / researcher.
Maxime Fournes
Karma: 365
Consider that the voting basis of the Republican party is in large majority anti-AI
Thanks! Will add
Crunch Time is Now. Drop Everything.
Epistemic status: Hopeful, stressed, terrified, ecstatic.
tl;dr This could be it. For the next few days, do nothing but contact journalists and policymakers (ideally via existing orgs), attend/share/organise protests near you, and use your public profile to speak out.
This is Our Chance
The explosion of interest in AI safety over the last week has been both inspiring and dizzyingly fast. We might actually win this thing.
Events since the Jacob Coxon resignation have blown the Overton window as wide open as it is likely to get. This is the kind of chance we have dreamed of, and we have a responsibility not to waste it. We may not get another.
What You Can Do
This window will not last forever. We need to act over a matter of days, not weeks or months.
This is the time to call in favours, work overtime, and put all your strength of heart and brain into keeping our world safe.
If you work in AI safety, we think the most impactful thing for you to do is to drop everything you are doing and pour your energy into policy and media outreach.
Politicians and journalists want to be able to speak on this issue but are keenly aware of their lack of technical knowledge. Cold-emailing and cold-calling them and offering to brief them has a surprisingly high likelihood of success.
This especially includes conservative ones. We don’t care what your personal politics are, survival is a non-partisan issue.
There are many ways in which people of all skillsets and levels of experience can contribute, but we suggest three key actions:
Contact journalists and members of congress/parliament, This is best done by volunteering for an existing organisation. We are capacity constrained and need your help. You do not need to be an expert: there are plenty of jobs that need doing.
Any other group you have a line into
Share and attend protests in your area and invite your friends and family. For instance:
Speak out: Twitter, Instagram, wherever. Be grounded in facts, but also convey the right mood. Also, we’re in the Overton window—speak accordingly.
We need to put out clear, actionable asks for policymakers to translate this momentum into real solutions. We suggest:
US—The people really want Donald Trump to negotiate a pause treaty with Xi Jinping and he might get a Nobel Peace Prize if he makes the deal. National security depends on it.
UK—Andy Burnham needs to announce that the UK will convene a pause treaty summit and thereby protect its citizens and honour its legacy of global influence and world-leading safety talent (e.g. Bletchley Park Summit, UK AISI).
Elsewhere—Every government can act within their own borders and put pressure on the US government to get on board. We cannot let unaccountable US tech giants put us all at risk.
Good Luck and Godspeed
For the first time in a very long time, we actually think things might go well for humanity. Whether that happens is up to us.
See you on the other side.
Maxime Fournes’s Shortform
Hi @nem, thanks for stepping up! We will try to come back to you ASAP on this
Thank you for this beautiful and heartfelt testimony. Know that many of us are working tirelessly to save what can be. It took me more than 30 years before I finally snapped out of cynicism and realised that single individuals can make a huge positive impact on the course of history. I invite you to speed-run this process and join us before it’s too late, we can use your help. We can use so much more help. Best of luck, and take care.
Thanks for the comment. Let me try to unpack.
On the statement comparison:
If you are thinking in term of raw number of signatures, I think you are conflating low reach with low reputation. We do have lower reach than FLI, but we are not operating with anywhere near the same resources. FLI 2024 total expenditures were $17M.
If you are thinking in terms of number of big names, there is also a question of reach. Our network is not as developed, again coming down to a matter of available resources. But there is also a selection effect. People like Bengio deliberately keep some distance from PauseAI advocacy to protect a middle-ground position, which suppresses some of our visible support. I guess you could call that advocacy being lower-status than research. That being said, I think it’s very important for us to maintain a good reputation and we continue working on it.
On School for Moral Ambition / ControlAI: I think these are doing a different work than us, and shallower—and I don’t mean shallow in a pejorative sense here but descriptive: getting a lot of people to sign a statement or to subscribe to your newsletter is the visible/top layer. What we are building is the thing underneath, the deep organised buy-in, individuals who start owning the issue and moving their social network.
It’s also worth noting again that Bregman started with a large pre-existing following, and that ControlAI’s operating budget is an order of magnitude above ours. We see ourselves as complementary with these organisations.And to bounce on your CeSIA survey: to me it shows that the latent constituency is already here, so the next big move is to organise it.
Thanks Ondrej! Well I would say that this is us being transparent about our funding situation right now, including internally. That said, we’re careful not to make people more worried than the situation warrants and demotivate them unnecessarily. We are working hard on fundraising right now and I am confident that we will unlock sufficient funding in time to continue operating. In fact, we just got word of a significant donation that should increase our runway by two months.
We will continue being transparent as things develop and if it really starts looking like we are about to run out, we will activate a contingency plan to return to a volunteer-driven structure.
On the doubts on “The right terminal goal is a binding international pause treaty with verification/enforcement”
I want to emphasise the instrumental framing of the pause. The real product of our work is a large constituency that groks catastrophic / xrisks, and that’s net-positive across a wide range of good policies, not just a pause: red lines, creating an international AI agency, verification mechanisms, the capacity to pause, etc. We organise around “pause” because:a movement needs one clear, concrete, defensible demand to cohere around. It has to stay simple and legible.
It also is a strong mechanism against capture by adversarial political actors (which also partially answers @TsviBT’s comment). It is much harder for the movement to get captured when it was built around a precise ask
if a large movement exists, it should be pushing for the most foolproof policy. And I believe that the pause is the policy that makes the most sense, that it’s what the most sane timeline would be doing when handling such a dangerous technology.
On “more palatable ways to present this to policymakers”, I want to remind that in our model the “dealing with policymakers at scale” part is handled by different orgs. Tailoring the pitch is mostly the policy orgs job (the Reformer role in the post) and we are happy to defer to them on it as long as we share a similar enough world model.
Thanks a lot, this is very valuable. There is a LOT to unpack, but I’ll do my best to answer what I think are the most important / strongest objections. And thanks @Benjamin Schmidt for replying, I overall agree with your responses.
I will start with the “Fable ban” one. First, I want to concede that “policymakers won’t make any move without a public constituency” is too strong a claim, and if that’s what you read, just know that this was not our intention to make it.
Here is the narrower claim that we are making, and I don’t think the Fable episode really makes a dent in it:
The natural incentive gradient does not currently point toward a binding, durable, internationally coordinated pause (or equivalent strong beneficial xrisk policy). That specific policy constrains both a powerful industry and the government’s perceived supremacy interests, and it needs sustained commitment plus international coordination. Without a counterforce, even sympathetic policymakers would stay put on that task. A one-off executive ban tells us about that ban’s incentive landscape: geopolitics, security framing, the ideology of the administration, etc. And our overarching point still applies: concentrated interests organise themselves, whereas diffuse public interests have to be organised.
Another way I would put it: the Fable episode is the output of an incentive structure, and right now those incentives are concentrated in a few powerful actors. If you want to change outcomes, you can create counterincentives in at least two places: you can push at the top of the pyramid, a handful of senior people in government, or you can push at the bottom/middle layers, which is what we are trying to do. A swing at the top is more effective, but the top is also very hard to steer: powerful actors have strong mechanisms for resisting being steered, and far more money and access has already gone into steering them (mostly by the industry). The middle and bottom layers are neglected by comparison. By building a strong movement pushing in the right direction, we increase the odds that shifts in policy go in the right direction, rather than leaving them at the mercy of whoever has the president’s ear that month.
There is also a point about robustness. A shift that lives only at the top is fragile: what one executive can do overnight without a constituency, another one can undo overnight without one. The change we are ultimately trying to create is deeper and harder to reverse, closer to shifting the culture / shifting norms about what’s acceptable, and that’s the kind of thing that holds across administrations. We had to cut that argument from the post for length, but it’s central to the ambitious version of our plan.
Existential AI safety needs an effective social movement. PauseAI is building it
Agreed, and especially true during carbon monoxide poisoning. During this event I was the one dialling the emergency, but I could hardly explain what was going on to the person on the phone. I barely had any thoughts anymore, and was going at a pace of about one word per second. When the lady asked me if there was a smell in the room, I just could not parse the question and I resorted to repeating it to the people around me and eventually gave the phone to Elisa.
Basics of How Not to Die
The argument overall proves too much about corporations
Does it? Aren’t corporations the ones building ASI right now?
Yes, it sounds like a real inflection point in acceleration to me...
I disagree with this. I would be happy to see more people freaking out about AI, especially mainstream medias. I would actually argue that getting more people to freak out as fast as possible is our best shot at lowering the odds of an extinction event.
Amazing energy people, that’s what we need 👍