CEO of PauseAI. Ex AI engineer / researcher.
Maxime Fournes
Karma: 274
Thanks for the comment. Let me try to unpack.
On the statement comparison:
If you are thinking in term of raw number of signatures, I think you are conflating low reach with low reputation. We do have lower reach than FLI, but we are not operating with anywhere near the same resources. FLI 2024 total expenditures were $17M.
If you are thinking in terms of number of big names, there is also a question of reach. Our network is not as developed, again coming down to a matter of available resources. But there is also a selection effect. People like Bengio deliberately keep some distance from PauseAI advocacy to protect a middle-ground position, which suppresses some of our visible support. I guess you could call that advocacy being lower-status than research. That being said, I think it’s very important for us to maintain a good reputation and we continue working on it.
On School for Moral Ambition / ControlAI: I think these are doing a different work than us, and shallower—and I don’t mean shallow in a pejorative sense here but descriptive: getting a lot of people to sign a statement or to subscribe to your newsletter is the visible/top layer. What we are building is the thing underneath, the deep organised buy-in, individuals who start owning the issue and moving their social network.
It’s also worth noting again that Bregman started with a large pre-existing following, and that ControlAI’s operating budget is an order of magnitude above ours. We see ourselves as complementary with these organisations.And to bounce on your CeSIA survey: to me it shows that the latent constituency is already here, so the next big move is to organise it.
Thanks Ondrej! Well I would say that this is us being transparent about our funding situation right now, including internally. That said, we’re careful not to make people more worried than the situation warrants and demotivate them unnecessarily. We are working hard on fundraising right now and I am confident that we will unlock sufficient funding in time to continue operating. In fact, we just got word of a significant donation that should increase our runway by two months.
We will continue being transparent as things develop and if it really starts looking like we are about to run out, we will activate a contingency plan to return to a volunteer-driven structure.
On the doubts on “The right terminal goal is a binding international pause treaty with verification/enforcement”
I want to emphasise the instrumental framing of the pause. The real product of our work is a large constituency that groks catastrophic / xrisks, and that’s net-positive across a wide range of good policies, not just a pause: red lines, creating an international AI agency, verification mechanisms, the capacity to pause, etc. We organise around “pause” because:a movement needs one clear, concrete, defensible demand to cohere around. It has to stay simple and legible.
It also is a strong mechanism against capture by adversarial political actors (which also partially answers @TsviBT’s comment). It is much harder for the movement to get captured when it was built around a precise ask
if a large movement exists, it should be pushing for the most foolproof policy. And I believe that the pause is the policy that makes the most sense, that it’s what the most sane timeline would be doing when handling such a dangerous technology.
On “more palatable ways to present this to policymakers”, I want to remind that in our model the “dealing with policymakers at scale” part is handled by different orgs. Tailoring the pitch is mostly the policy orgs job (the Reformer role in the post) and we are happy to defer to them on it as long as we share a similar enough world model.
Thanks a lot, this is very valuable. There is a LOT to unpack, but I’ll do my best to answer what I think are the most important / strongest objections. And thanks @Benjamin Schmidt for replying, I overall agree with your responses.
I will start with the “Fable ban” one. First, I want to concede that “policymakers won’t make any move without a public constituency” is too strong a claim, and if that’s what you read, just know that this was not our intention to make it.
Here is the narrower claim that we are making, and I don’t think the Fable episode really makes a dent in it:
The natural incentive gradient does not currently point toward a binding, durable, internationally coordinated pause (or equivalent strong beneficial xrisk policy). That specific policy constrains both a powerful industry and the government’s perceived supremacy interests, and it needs sustained commitment plus international coordination. Without a counterforce, even sympathetic policymakers would stay put on that task. A one-off executive ban tells us about that ban’s incentive landscape: geopolitics, security framing, the ideology of the administration, etc. And our overarching point still applies: concentrated interests organise themselves, whereas diffuse public interests have to be organised.
Another way I would put it: the Fable episode is the output of an incentive structure, and right now those incentives are concentrated in a few powerful actors. If you want to change outcomes, you can create counterincentives in at least two places: you can push at the top of the pyramid, a handful of senior people in government, or you can push at the bottom/middle layers, which is what we are trying to do. A swing at the top is more effective, but the top is also very hard to steer: powerful actors have strong mechanisms for resisting being steered, and far more money and access has already gone into steering them (mostly by the industry). The middle and bottom layers are neglected by comparison. By building a strong movement pushing in the right direction, we increase the odds that shifts in policy go in the right direction, rather than leaving them at the mercy of whoever has the president’s ear that month.
There is also a point about robustness. A shift that lives only at the top is fragile: what one executive can do overnight without a constituency, another one can undo overnight without one. The change we are ultimately trying to create is deeper and harder to reverse, closer to shifting the culture / shifting norms about what’s acceptable, and that’s the kind of thing that holds across administrations. We had to cut that argument from the post for length, but it’s central to the ambitious version of our plan.
Existential AI safety needs an effective social movement. PauseAI is building it
Agreed, and especially true during carbon monoxide poisoning. During this event I was the one dialling the emergency, but I could hardly explain what was going on to the person on the phone. I barely had any thoughts anymore, and was going at a pace of about one word per second. When the lady asked me if there was a smell in the room, I just could not parse the question and I resorted to repeating it to the people around me and eventually gave the phone to Elisa.
Basics of How Not to Die
The argument overall proves too much about corporations
Does it? Aren’t corporations the ones building ASI right now?
Yes, it sounds like a real inflection point in acceleration to me...
I disagree with this. I would be happy to see more people freaking out about AI, especially mainstream medias. I would actually argue that getting more people to freak out as fast as possible is our best shot at lowering the odds of an extinction event.
Thank you for this beautiful and heartfelt testimony. Know that many of us are working tirelessly to save what can be. It took me more than 30 years before I finally snapped out of cynicism and realised that single individuals can make a huge positive impact on the course of history. I invite you to speed-run this process and join us before it’s too late, we can use your help. We can use so much more help. Best of luck, and take care.