Another perhaps interesting observation is that for the first ~20 years of AI safety (i.e., until “Death with Dignity” in 2022), the field was almost fully united behind this “try to build safe ASI, don’t bother trying to coordinate a pause” strategy.[1] This seems remarkable both for itself (how to explain the lack of diversity of thought on this topic?) and for the lack of anyone noticing or investigating this anomaly until now.
Aside from my own posts, the closest talk or writing I can remember or find that’s in favor of a pause-like position before 2022 is Brian Tomasik’s 2013 International Cooperation vs. AI Arms Race but he wasn’t that enthusiastic about it: “I suspect more direct MIRI-type research has higher expected value, but among EAs who don’t want to fund MIRI specifically, encouraging donations toward international cooperation could be valuable, since it’s certainly a more mainstream cause.”
Does anyone remember anything else? I tried several AI-powered queries/searches, but don’t fully trust the negative results since it’s hard to be sure what keywords to use, and maybe some materials have fallen off the indexed web.
To echo @StanislavKrym it seems like there were a few assumptions that pretty easily explain the lack of diversity around this topic.
1. That compute power would not be a bottleneck. (Right now, the means of trying to stop AGI from being built is largely via compute power restriction; if it were not a significant bottleneck, then global agreements about this would be immensely more intrusive and dubious.)
2. That FOOM would be fast, and we would not have multi-year long period where people can actually talk to AI but we have no superintelligence; this kind of period is one of the reasons a “pause” is plausible. (In “There is No Fire Alarm” for instance Yud says that people will only think AGI is imminent when “the AI seeming pretty smart in interaction and conversation; aka the AI actually being an AGI already.” Really that whole article is about how people are going to just not realize things will happen till they happen; it is actually premised on a much faster FOOM.)
3. That there would be “infohazards” about AI, potentially related to AI creation, which making generally known would be enormously net negative. (I.e., MIRI actually stopped publishing because they thought they might have such infohazards. If you’re concerned about drawing attention to these things than an “International Pause” is again plausibly the worst thing you could do, because you’re drawing a bunch of attention to a small space.)
The third seems a bit fuzzier and worse as reason not consider a pause, although I wouldn’t be surprised if it was one such reason. The first two seem to me pretty compelling reasons this option would not be considered. In general, I expect all of these were operating not as explicit considerations but as general hidden steering vectors, keeping this option from being available.
Paul Christiano didn’t have these beliefs/assumptions (and was prominent on LW), and more and more people in AI safety had Paul-like views as DL became more popular/successful through the 2010s (including safety people at DeepMind, founded in 2010, and OpenAI, founded in 2015), but still no significant Pause position or advocacy appeared until 2022.
There is still no firealarm for ASI. How things will work out is still uncertain. The AI talking to you is more like smoke than a firealarm in the firealarm analogy. It’s still hard to coordinate a pause around. But yeah past ChatGPT your chances are still higher than post GPT-2 or post AlphaGo. I don’t know where I read it but yeah I think I read somewhere from some people involved in Miri explicitly that they are not trying to make the wider public pay too much attention to AI which makes complete sense given how people end up doing stupid things in response to that (the founding of OpenAI etc.). So not making too much of a fuss at that time just seems correct.
Anonymous: I’m curious if the grim outlook is currently mainly due to technical difficulties or social/coordination difficulties. (Both avenues might have solutions, but maybe one seems more recalcitrant than the other?)
Eliezer Yudkowsky: Technical difficulties. Even if the social situation were vastly improved, on my read of things, everybody still dies because there is nothing that a handful of socially coordinated projects can do, or even a handful of major governments who aren’t willing to start nuclear wars over things, to prevent somebody else from building AGI and killing everyone 3 months or 2 years later. There’s no obvious winnable position into which to play the board.
Anonymous: just to clarify, that sounds like a large scale coordination difficulty to me (i.e., we—as all of humanity—can’t coordinate to not build that AGI).
Eliezer Yudkowsky: I wasn’t really considering the counterfactual where humanity had a collective telepathic hivemind? I mean, I’ve written fiction about a world coordinated enough that they managed to shut down all progress in their computing industry and only manufacture powerful computers in a single worldwide hidden base, but Earth was never going to go down that route. Relative to remotely plausible levels of future coordination, we have a technical problem.
@Wei Dai IIRC Yudkowsky made the (erroneous) assumption[1] that mankind could’ve started thousands of AGI projects. If building the AGI was as easy as constructing a cartel and trading drugs, then mankind would’ve zero hope to prevent this and would be able only to ensure that a GOOD actor builds it FIRST.
What do you think “coordinate a pause” would have looked like in a pre-AI-race world? Most current pausers (most explicitly FLI) want to pause AI at current rate of capabilities, and even then only the kind of agentic AI capabilities research that tend toward building AGI/ASI, while supporting development of narrow Tool AI. But the person most associated with defending the Tool AI over Agentic AI position in early 2010s LW discourse (@HoldenKarnofsky) is also the one who definitely more than anyone else mentioned here is responsible for the AI race happening at all.
What do you think “coordinate a pause” would have looked like in a pre-AI-race world?
be more humble/uncertain about personally, organizationally, or civilizationally solving AI alignment/safety
talk about alignment/safety research potentially failing and having to coordinate a pause
don’t make plans/claims about unilaterally (i.e., without widespread agreement/consent) building a world-changing Friendly AI or Pivotal Task AI, thereby normalizing the idea and spreading it to others (and try to discourage such plans in others)
start/encourage LW discussions about politics, social activism/movements, large-scale coordination, in theory and practice
think ahead about and encourage discussions about potential strategies and obstacles to coordinating a pause
try to convince prominent AI researchers of AI x-risks and get them to spread the message to the public earlier
gather resources (people, funding, connections, etc.) and build institutions for a potential pause effort
depending on how these efforts go, do one or more of:
try to pre-emptively ban/regulate AI development
start an international safe AGI/ASI project while banning others
wait for an AI arms race to start and then oppose it from a well-prepared position
So, I think (haven’t read it in nearly a decade) Superintelligence did do 2 and 5 and push toward 8b has the ideal solution? MIRI and FLI did do 6 (e.g. Stuart Russell on the board, 2015 letter), 7, and eventually 8c obviously. I think that 8a being incoherent, and the main defenders of 1-3 being the ones to actually launch the AI race, was my main point here.
Another perhaps interesting observation is that for the first ~20 years of AI safety (i.e., until “Death with Dignity” in 2022), the field was almost fully united behind this “try to build safe ASI, don’t bother trying to coordinate a pause” strategy.[1] This seems remarkable both for itself (how to explain the lack of diversity of thought on this topic?) and for the lack of anyone noticing or investigating this anomaly until now.
Aside from my own posts, the closest talk or writing I can remember or find that’s in favor of a pause-like position before 2022 is Brian Tomasik’s 2013 International Cooperation vs. AI Arms Race but he wasn’t that enthusiastic about it: “I suspect more direct MIRI-type research has higher expected value, but among EAs who don’t want to fund MIRI specifically, encouraging donations toward international cooperation could be valuable, since it’s certainly a more mainstream cause.”
Does anyone remember anything else? I tried several AI-powered queries/searches, but don’t fully trust the negative results since it’s hard to be sure what keywords to use, and maybe some materials have fallen off the indexed web.
To echo @StanislavKrym it seems like there were a few assumptions that pretty easily explain the lack of diversity around this topic.
1. That compute power would not be a bottleneck. (Right now, the means of trying to stop AGI from being built is largely via compute power restriction; if it were not a significant bottleneck, then global agreements about this would be immensely more intrusive and dubious.)
2. That FOOM would be fast, and we would not have multi-year long period where people can actually talk to AI but we have no superintelligence; this kind of period is one of the reasons a “pause” is plausible. (In “There is No Fire Alarm” for instance Yud says that people will only think AGI is imminent when “the AI seeming pretty smart in interaction and conversation; aka the AI actually being an AGI already.” Really that whole article is about how people are going to just not realize things will happen till they happen; it is actually premised on a much faster FOOM.)
3. That there would be “infohazards” about AI, potentially related to AI creation, which making generally known would be enormously net negative. (I.e., MIRI actually stopped publishing because they thought they might have such infohazards. If you’re concerned about drawing attention to these things than an “International Pause” is again plausibly the worst thing you could do, because you’re drawing a bunch of attention to a small space.)
The third seems a bit fuzzier and worse as reason not consider a pause, although I wouldn’t be surprised if it was one such reason. The first two seem to me pretty compelling reasons this option would not be considered. In general, I expect all of these were operating not as explicit considerations but as general hidden steering vectors, keeping this option from being available.
Paul Christiano didn’t have these beliefs/assumptions (and was prominent on LW), and more and more people in AI safety had Paul-like views as DL became more popular/successful through the 2010s (including safety people at DeepMind, founded in 2010, and OpenAI, founded in 2015), but still no significant Pause position or advocacy appeared until 2022.
There is still no firealarm for ASI. How things will work out is still uncertain. The AI talking to you is more like smoke than a firealarm in the firealarm analogy. It’s still hard to coordinate a pause around. But yeah past ChatGPT your chances are still higher than post GPT-2 or post AlphaGo. I don’t know where I read it but yeah I think I read somewhere from some people involved in Miri explicitly that they are not trying to make the wider public pay too much attention to AI which makes complete sense given how people end up doing stupid things in response to that (the founding of OpenAI etc.). So not making too much of a fuss at that time just seems correct.
I had a similar reflection in this Jan 2025 review of the Shut it All Down letter. In that thread I referenced Discussion with Eliezer Yudkowsky on AGI interventions, where the pause advocate was anonymous:
Good job anonymous questioner of November 2021. But that’s not much earlier than the 2022 sources you found, and it’s just asking questions. The other pause advocacy I found at the time was 2022 era: What an Actually Pessimistic Containment Strategy Looks Like in April 2022 and Let’s Think About Slowing Down AI in December 2022.
@Wei Dai IIRC Yudkowsky made the (erroneous) assumption[1] that mankind could’ve started thousands of AGI projects. If building the AGI was as easy as constructing a cartel and trading drugs, then mankind would’ve zero hope to prevent this and would be able only to ensure that a GOOD actor builds it FIRST.
I expect the creatuion of AGIs as efficiently as the human brain to be bottlenecked not on compute, but on our lack of technology.
What do you think “coordinate a pause” would have looked like in a pre-AI-race world? Most current pausers (most explicitly FLI) want to pause AI at current rate of capabilities, and even then only the kind of agentic AI capabilities research that tend toward building AGI/ASI, while supporting development of narrow Tool AI. But the person most associated with defending the Tool AI over Agentic AI position in early 2010s LW discourse (@HoldenKarnofsky) is also the one who definitely more than anyone else mentioned here is responsible for the AI race happening at all.
Edit: I think @Eliezer Yudkowsky’s Six Dimensions of Operational Adequacy in AGI Projects, a 2017 MIRI internal document intended as a criticism of the Karnofsky-led OpenPhil/EA approach to AI labs, is, characteristically, the first expression of MIRI’s pro-pause stance. So my conclusion is that the pro-pause stance only really makes sense if there’s already an AI arms race to stand in opposition to. (Mind you, Tomasik talks only of fostering general international cooperation and expecting lower AI risk as a positive flow-through effect.)
be more humble/uncertain about personally, organizationally, or civilizationally solving AI alignment/safety
talk about alignment/safety research potentially failing and having to coordinate a pause
don’t make plans/claims about unilaterally (i.e., without widespread agreement/consent) building a world-changing Friendly AI or Pivotal Task AI, thereby normalizing the idea and spreading it to others (and try to discourage such plans in others)
start/encourage LW discussions about politics, social activism/movements, large-scale coordination, in theory and practice
think ahead about and encourage discussions about potential strategies and obstacles to coordinating a pause
try to convince prominent AI researchers of AI x-risks and get them to spread the message to the public earlier
gather resources (people, funding, connections, etc.) and build institutions for a potential pause effort
depending on how these efforts go, do one or more of:
try to pre-emptively ban/regulate AI development
start an international safe AGI/ASI project while banning others
wait for an AI arms race to start and then oppose it from a well-prepared position
So, I think (haven’t read it in nearly a decade) Superintelligence did do 2 and 5 and push toward 8b has the ideal solution? MIRI and FLI did do 6 (e.g. Stuart Russell on the board, 2015 letter), 7, and eventually 8c obviously. I think that 8a being incoherent, and the main defenders of 1-3 being the ones to actually launch the AI race, was my main point here.