I often hear people say they think we should pause AI at some point, but not yet. Their basis for this seems to be some combination of:
If we pause at the last possible moment, then we will have the most advanced AI possible during the pause, which will be helpful for doing AI safety research during the pause
Implicitly, there is some quantity of ‘pausing credit’, that will buy us a few months of pause say, and if we use them now, we won’t have them to use later, when it is important
If we pause, and then AI doesn’t seem to be at dire risk of destroying the world, maybe the public will backlash against this and it will be harder to do any kind of AI safety (especially if it has major economic consequences)
The models aren’t dangerous yet
This all sounds very questionable to me. I suggest instead that the following are at least as likely to be true:
We can’t pause on a dime at the precise second that ‘we’ decide it is important to—pulling the breaks will take a while, during which time we will continue to rocket into danger.
If we managed to pause now, that would greatly increase the chance we paused again later. At the moment, a major obstacle to people supporting this is their senses that it is impossible. Building the machinery to pause makes it much easier to pause again. Doing something once is in general extremely helpful for doing it later times—if you are taking the most important action in the world, you really don’t want to be trying it for the first time, when it matters.
The public substantially hates AI, but feels incredibly disempowered, because for instance they buy the story that technological ‘progress’ is inexorable. If they saw vividly how much power our institutions have to shape the course of technology, they would become more activated against AI.
Some of the models seem to be some amount dangerous, we can’t tell how dangerous the new ones are, they are improving very fast, and the point where we are confident that they are currently dangerous is very suboptimally late to start this project.
“Doing something once is in general extremely helpful for doing it later times” etc
It depends. Given the politics of anti-pandemic measures, I am not sure that we are as “prepared” against a pandemic as in 2020.
That’s a good question! One big difference is that the most stringent measures against covid, eg lockdowns, had annoying consequences for the general population, thus producing a backlash if not seen basically as the only option (my impression is that past the very first lockdown, where the uncertainty scared people enough, they became broadly unpopular)
My 2 comments:
1) Pausing AI research would cause minimal disturbance on the general population. It would be one of those abstract political questions some people would overstate the benefits/harm of, without big concrete observable effects for most (actually ~all) people. (this doesn’t contradict your comment which made the point that there might be counter-examples to the statement “Doing something once is in general extremely helpful for doing it later times”)
2) I’m not convinced we are at all “less prepared” than in 2020. It’s quite unlikely the next serious pandemic is very similar to covid in terms of infectiousness profile and severity profile, so people would adapt differently depending on these parameters, and not ignore it because “covid was fine” (eg imagine if it’s more dangerous to children this time). At least I would expect the public to take it more seriously in the early phase, unlike covid which was downplayed until too late to ignore. We also still have the memories of how to adapt our life to different restrictions, and that we can adapt fast. The rate of people being “science skeptical” seems to be roughly the same as before 2020, from the empirical research I saw (epistemic confidence: low)
Well what would you say it depends on? Part of the COVID response was that we got lucky and the virus wasn’t as bad as it could have been. Had it been worse, the general response could have been better.
Here, it seems like the public sentiment is that AI is a net negative and that reacting to it would be a good option. If a good portion of the public agrees that we’ve pulled something much more dangerous than COVID out of the urn, how could this be analogous?
And then there’s the fact that if we have to Pause again, AI will almost necessarily be more dangerous. If post-COVID we know that the next virus will be worse, and potentially much worse, then COVID probably did offer some good degree of prep.
Whether “doing something once” is “extremely helpful for doing it later times” depends on whether people agree—while it is done and afterwards—on whether the measure is/was good, effective, fair, and whether people think the benefits exceeded the costs. It seems relevant whether people believe that it worked well and whether people more or less understood what worked and what didn’t.
In the abstract, I assume that people agreed before 2020 that anti-pandemic measures can be necessary. When the pandemic came and they were enacted, there was less consensus. Additionally, more people seem to evaluate the measures negatively in retrospect. (Personal impression, not a data-based statement.)
During Covid, there was political opposition to the measures, but some years later, this opposition has probably had more time to coordinate and develop.
I’m not sure I’d agree with your definition of success in trial runs. First, because the average response to how we handled COVID won’t generalize to viruses that are substantially more dangerous. If AI is paused and development continues, there will almost necessarily be larger risk in the larger capacity. If it’s substantially stronger than it was when it was initially paused, then it’s like comparing the COVID response to a hypothetical virus that’s visually much more dangerous—if it looked like the black death and there were videos of people passing out on the street because of it, I’m certain there would be a more prompt response.
Second, Operation Warp Speed was incredibly effective to my understanding in creating effective vaccines extremely quickly and then mass producing and distributing hundreds of millions of them. Using OWS as a model to improve from for the next pandemic seems like it could only help us.
I think it’s hard to make a list of when to tell that doing something once is helpful for doing them again, but some broad aspects would be A, the ability to learn from the first event, B lasting infrastructure or means of doing something as a result of the first event, C how well we’re able to test/understand A and B, the list goes on but I don’t think expanding it would necessarily be helpful
With your original comment, are you just saying that trial runs being generally helpful doesn’t always apply or are you saying that it wouldn’t apply in this instance and COVID is a strong indicator of that? If it’s the latter, can you give more reasons that the analogy works?
Great post! Agree with all of the above.
For folks who want to pursue the Pause ASAP case, about 110+ of us will be marching on OpenAI, Anthropic and Google on July 11, asking Sam Altman, Dario Amodei and Demis Hassabis to make public commitments about pausing frontier AI development (if others also pause).
Sign up here: https://luma.com/s0k8wvee
Or here: https://partiful.com/e/EChgbBMsoeN3qVnATHKJ
---
How I think protesting now helps us get to an AI Pause / International Treaty ASAP:
Coalition Building with other organisations that are already activated on other issues, and are starting to get worried about AI. Our July 11 protest will be part of a larger coalition (theprotest.ai). Right now it’s featuring 6 orgs, mostly AI Safety, but including The National Union of Healthcare Workers. We expect more orgs like the league of women voters to join next week.
Raising political salience through media attention. Last march was featured in major newspapers in the US (New York Times (1, 2), Washington Post, The Atlantic), MS NOW (prev MSNBC), and major news outlets in Brazil / Spain.
Pressuring the labs to making more conditional statements about pausing. After our protest, Anthropic wrote it expects it “would slow down or temporarily pause” if other labs verifiably did too, and OpenAI wrote it expects coordination, including slowing frontier development, to become more important. We have reasons (that we can only keep private for now) on why we believe our protest was instrumental for OpenAI, in particular for their first blogpost.
Putting an AI Pause at the center of the public discourse, so that AI Lab CEOs can’t escape it. This was what happened with the Google DeepMind Hunger Strike, where months later a journalist (Emily Chang) asked Demis Hassabis the same question we were asking him, which lead to him making a statement (or at least a comment / answer) on pausing AI if everyone else pauses.
Things that are not discussed in the above:
Concrete discussions of what an ideal pause would look like, and what the side effects of even an ideal pause might be. (Banning runs above N flops? Unilateral datacenter moratorium? No more higher scores on SWEBench? Etc etc)
Concrete discussions of the likely difference in implementation between an ideal pause (the one all us right-thinking people on LW would endorse) and the actual pause-resulting we would get would be like (oops there’s a carve-out so the NSA gets the most advanced AIs).
Back-casting of whether pushing even harder for a pause in 2023 would have been a good idea.
I think it would be good to talk about the conditions for lifting or extending the pause after it has been implemented. It’s not a top area of concern for me, but I just completed an AI Safety course with BlueDot where my facilitator disfavored a pause for this reason (in addition to political feasibility and monitoring concerns).
For common knowledge: I don’t feel like this really responds to where thoughtful people are coming from in part because it isn’t specific enough about the proposal or what people are claiming. E.g., what does “pause now” concretely mean (there are many possible things here!). TBC, I’m not claiming everyone is thoughtful or has reasonable objections and I broadly agree with most of the counterarguments you raise. (I just don’t think these arguments are sufficient to imply a very strong conclusion about what we should do.)
The post wasn’t really a fleshed out version of itself, but, it seemed to be less advocating for a particular kind of pause, and more advocating that various arguments for not pausing now don’t seem like good reasons.
I read it like “whatever sort of pause you thought might be a good idea later, do you actually have a good reason not to do it now?”
I am pretty worried that as we get closer towards RSI that progress will have some momentum of its own. We might already be past that point.
Then Dwarkesh makes a youtube video telling everyone to maxx sample efficiency. One man’s taboo is always another’s brilliant insight.
AI has solved quite a few open mathematical problems within the last couple of months. I wouldn’t rule out AI becoming dangerous very soon.
If we wait until AI can outperform humans at every conceivable cognitive task, by then it’s too late.
Also lobbying for a pause becomes considerably harder once the AI is capable of swamping the internet with persuasive well-written articles advocating against a pause.
The claudes are already used by my coworkers to convince management, used by management to convince customers, used by customer’s lawyers to redline contracts, used by pentesters for everything, used by applicants to get interviews, and all these things to great effect.
Anthropic and OpenAI have recently published statements on the topics of RSI / AI doing AI research and the need for a coordinated pause / slowdown. I’ll post one paragraph and link to the rest.
Anthropic:
OpenAI:
Is anybody aware of statements by other relevant actors I’ve missed?
https://www.anthropic.com/institute/recursive-self-improvement
https://openai.com/index/built-to-benefit-everyone-our-plan
Aren’t the controls on Fable evidence against this? Granted it takes time to pause AI “properly”, but if there’s a warning shot of some sort I can totally imagine the US shutting down domestic AI development very rapidly. It would take longer to bring China on board, but probably not so long that they would catch up to the frontier. Is the argument that we can’t pause on a dime without an explicit warning shot? I think the time it takes to implement a pause is largely about building the political will, so I buy the argument that we should be building political will now.
A “pause” on public deployment without a pause on further development is in many ways worse than useless.
I agree, but I think if there were political will the government could shut down development (at least at the major US labs) rapidly as well. As I said, a proper pause would take time to implement, but I think you could get like 80% of the way there by just telling the big US labs to stop development and working out the details later. Again, I’m not arguing that this is the best plan—obviously it would be better to have something properly implemented, and I agree with the post’s overall argument that we should pause ASAP. I just don’t think it actually takes much time to implement something basic.
The concern is that i) they’ll do that in an emergency, but it’s unclear that stopping further development could occur in time given that once it’s dangerous it’s too late, or ii) they’ll discuss doing a bam preventatively and this requires significant debate and implementation so it will be too slow.
What are the odds that we get a sufficiently scary warning shot that governments can rapidly coordinate to implement a pause in a short time? “We shouldn’t pause now, because maybe we will get a warning shot and then pause later” seems like a foolhardy plan to me.
I never said this was a good plan? I agree with the post’s overall argument that we should work on a pause ASAP. I just don’t think this particular argument—that it takes time to implement a proper pause—is very strong, since it wouldn’t take long to implement something basic and work out the details later.
I think a pause would take a long(ish) time in the mainline scenario:
Develop enough support for a pause among policy-makers and the public.
Write proposals for how to implement a pause; figure out how to implement the details (how to do GPU tracking etc.).
Set up negotiations between relevant countries.
Set up more negotiations when the first round in all likelihood goes nowhere.
The governments agree to implement a pause starting 2 years from the negotiations, because that’s how these things usually work.
We’re already several years deep into steps 1 and 2 and there’s still a long way to go.
The “there will be no warning shots” claim was always the security mindset talking: it could equally be phrased “we should be able to survive even if there are no warning shots” and “a rogue AI smarter than you has a strong incentive not to give you a warning shot but instead bide its time until it’s sure of sucess (unless that means you might meanwhile build something else smarter than it)”.
IMO, we currently have a fusillade of minor warning shots going on, and as people rush to add agents to the economy, we’ll get louder ones.
This comment seems correct to me. The recent government action is certainly evidence that the government can act quickly to restrain AI development. Obviously there could be future situations in which that is not the case, and effective, durable, pauses will be much harder. But I updated some from the recent events (and should have been pricing this in more than I was)
Curious as to why China will take longer than the USA. Is the thinking there that China is already behind so could gain a bit of ground before they need to take action (basically a scenario Katja is arguing against) or something else?
My point was just that if the US government freaks out and decides to pause AI development, it could do so very rapidly domestically, while working out a proper, long-term arrangement with China would take longer—you’d need to communicate the risks to them in a way they’ll trust, work out exactly how monitoring and verification will work, etc. But I think if there’s the political will, this won’t take so long that China will entirely catch up to the US, and the US can threaten to restart development if China doesn’t come to the table.
I think it’s common to not leave this implicit but to instead explicitly highlight that the limiting factor is the lead that you have over other actors who are not pausing.
Maybe you’re talking about a very broad international pause here, where this applies less. If so, it’d be helpful to say so explicitly since the considerations are pretty different.
(FWIW, at least I don’t know anyone who has the view that an international pause is feasible but that we can only afford to have one for a few months. “We can only pause for a few months” sounds like something people would say about a unilateral pause where others are a few months behind.)
A pause is impossible, and is also a dangerous governmental overstep. At best they can stop public usage, and that is certainly less in the public interest than proliferation. Labs won’t stop researching. Capital runs America at the end of the day, as we all know, and that will ultimately decide the direction we take. I’d say a more worthwhile use of thought is to ponder how to adapt and resist the current situation, rather than a fairy tale of a pause happening. It is probably the case that the only possible pause can come from AI itself
The labs themselves have communicated their desire for building a coordination mechanism that enables a pause. Even capitalists don’t want to die from losing control over RSI AI systems. I’m not sure what’s going on in the mind of military strategists though. They might prefer to die from their own weapons rather than those of their enemies.
Will the labs stop researching if Trump sends thousands of federal agents to the headquarters of all the labs and appoints a panel of bankruptcy judges to dissolve the labs and return their assets to their creditors and shareholders?
Perhaps a reasonable unboxing of the word “pause” here is: “let off the gas, tap the brakes, breathe, and course correct”. Undeniably there is much untapped with the current frontier that we most certainly could stop racing without catastrophic side effects, towards a fruitful narrower path heavy on the sciences.
Agree that a pause is unlikely. But, if that’s a fairy tale, then what else isn’t a hail mary? If by adapting and resisting you mean fully accepting near term unsurvivable catastrophe, then yeah, that’s a reasonable outlook.
Anything else is something you need to make a case for.
Haha no I think you’re right; that by my view it’s basically the coin flip of apocalyptopia
I’ll just add that pausing ASAP may also be pausing barely in time. International agreements usually take a painfully long time. Many of us are skeptical that a pause is possible; it seems likely it’s at least difficult and therefore time consuming.
And the best-developed predictions of time to takeover-capable AI have distributions with substantial weight down to two years. (IMO of course; but it seems reasonable to take predictions like the AI futures with a lot of time on task more seriously than guesses from AI experts who’ve thought less about the mechanistic path to TCAI)
I also think there’s another important reason most AI safety people haven’t favored a pausing asap (up to now); Motivated reasoning. Advocating for a pause is difficult work we’re not skilled at and largely won’t enjoy. And most of us actually love AI and look forward to the help we’ll get from the next better generation.
I think the rest of the logic is also substantially correct. I think pausing ASAP is pretty obviously a good thing to advocate for; the only question is how much of our efforts should go toward that advocacy.
Right at the moment of the post, the mythos export control issue is unresolved, creating uncertainty for work at the frontier. With hindsight, we may look back at this period and see it as a pause, at least for deployment.
Is this pause being usefully exploited?
Is a pause in deployment actually helpful for safety?
I think it’s roughly net neutral. It reduces incentives to develop but not much, and it reduces outside safety scrutiny of internal systems by a lot.
As soon as we get ASI we are all dead—so honestly it just maybe gives us a bit more time on this earth
The idea that alignment of an ASI is a tractable problem is obviously absurd
I’m personally against a pause for a lot of reasons. I also think that the US government request to delay gpt5.6 is consistent with my belief that we are presently in a pause.
I further assert that as this pause continues, nothing of value for AI safety will be done. I expect pro-pause advocates will agitate first for ‘opening the labs to scrutiny’ then ‘stopping development in addition to deployment’, and eventually ‘making it permanent’. All without producing any economic value, or useful contributions to the fields where better LLMs would have helped.
But, we got a pause, so I hope someone proves me wrong.
I think that a “bad” pause, where we pause despite current/immediate risk being low, which e.g. causes China to catch up if it’s unilateral, or which powerful special interests (probably rightly) point out that it doesn’t do much to immediately reduce risk, could backfire and reduce the likelihood of a pause in the future. I overall currently feel basically 50-50 on whether pausing “early” increases or decreases the likelihood of a later pause.
I’d like to question the premise that the public hates AI. I think people have split preferences on it—they don’t want the world to end and also welcome some of the benefits of easier access to knowledge. I think there is a reason why AI apps have been at the top of the appstore charts for years now
Agreed with a lot of that. Note that Trump stopping AI deployments might be one of his most popular moves politically. It’s not hard to see a pause of capabilities development being very popular.
I would generalize this because some people say things like “bad regulation can easily make things worse”, and though it is not clear to me what the author of the post I link to means with that warning, I note that it would be an unrealistic expectation that we end up with exactly the right amount of regulation. Some people warn against AI regulation by referring to nuclear regulation. But it may be necessary to balance the possibilities of “bad overregulation” vs “bad underregulation”, both within a set of realistic policies.
What is extraordinary is there is no major politician of any country, that I’m aware of, stating that both AI risk is the key global priority and also that we need to pause
That should surely be the thing we need to change and anyone with clout needs to be trying to bend the ear of governments.
I don’t understand a p(doom) less than 95%, people can’t even explain to me a coherent story of how non—catastrophe happens. And yet there is no traction with world leaders at all.
Moreover, I’d say that stopping “at the last possible moment” and other arguments implicitly says a lot of things, all of which I deem to be 10% likely:
We know when the last possible moment even is. To be able to say that, is to say that 50% of ai research has succeeded. What I mean by that is that for any group to say “Model version N is at a position such that Model version N+1 will destroy civilization”, requires a complete understanding of some Model N, which is presumably some N-x versions away in the future, and thus more complex and more advanced than what we have today.
We know whether a model isn’t dangerous. To know whether a model isn’t dangerous implies that we can know whether a model is dangerous. This requires us, again, to have progressed far into ai research in a distance there is no evidence to claim is possible or attainable
We know whether a model is dangerous, such that we need to pause ai development, and therefore will not release the model to the public. This implies that we have the ability to contain a model that is very capable. This ability to contain these models, which has been discussed a lot, has not been shown to exist.
The above claims do not refer to the models currently blocked by the US government. Personally, I’d say with 99% confidence that the capabilities that are required to end civilization do not exist in those models.
The biggest reason why to pause ASAP is that the further the precipice we pause, the more wiggle room there is for rogue actors to break the pause and be stopped without dooming everyone else or advancing capabilities in a way that cannot be reversed and put us in riskier states.