My own bias is that if we can pause, we should. I will be quite happy if the future proves me wrong about this. But if we cannot pause, then I desire to believe that we cannot pause. (https://www.lesswrong.com/w/litany-of-tarski)
I do think this is a really invaluable analysis. I think if we are going to pause, our best bet is to rally around something like this.
There’s a huge bipartisan outcry against AI right now, and giving that a concrete rallying point would be really powerful—but that crowd speaks a very different language, and has very different concerns. I think if you really want a pause, you might want to consider pivoting just enough to try and appeal to that crowd—if you can get the artists and writers and displaced programmers behind you, I think this has a much better chance of success.
Personally, my biggest concern with a pause is the risk of dispersal—it’s a lot easier to control two frontier labs, and Anthropic has been an unusually responsible steward compared to most companies and governments.
Objection 1 - Fast Timelines
AI 2027 and Situational Awareness seem to converge on the idea that we could see something like ASI by 2030. That gives us four years. International politics move very slowly—getting a pause in place within two years seems like a political miracle to me, and even three years feels like a stretch. In terms of these timelines, we’re talking about slamming on the brakes. Moving slowly risks missing the window of opportunity.
This is less an objection to Pausing, and more skepticism about the viability
Objection 2 - The Economy
Right now the entire US economy is a massively leveraged bet on AI capabilites continuing to improve. If we slam on the brakes, there’s a good chance the bubble pops, resulting in massive economic turmoil. This means that a lot of people have very good reasons to oppose a pause, and whatever politician implements it might be committing career suicide.
I’d personally be willing to pay this cost, but again, it increases my skepticism about how much buy-in you can actually get.
Objection 3 - Acceleration
The other side of Economy is the profit motive: if some subset of capabilities are blocked, people can still work on making current models cheaper, more efficient, and building an ecosystem of harnesses and tools that let us get more out of existing capabilities. The more you crack down on this, the more you amplify the economic damage of a pause—at the extremes you risk the extinction of the entire industry.
These same motivations apply to academics: people still need to run studies and publish papers to survive, so they’re going to keep doing that. Even if you crack down on publication, a lot of research is still going to be done in private, in anticipation of the pause lifting.
Both of these mean that the instant a pause drops, progress suddenly surges forward. The next model can be trained around a more powerful ecosystem, and incorporate a huge corpus of previously unpublished work. Academics suddenly jumps forward to make up for the missing years.
This is where I start to be concerned about whether a Pause is actually beneficial.
All of this also means that every precaution built during that pause has to stand up to the returned weight of exponential progress *and* a sudden pent-up surge. On top of that, hardware and other more basic capabilities will have advanced, so exponential progress again jumps forward as training runs suddenly skip a step.
Objection 4 - Aligning the Unknown
It’s hard to anticipate what a 10x model will be capable of. It’s a lot easier to solve problems once you know what they are. Trying to build alignment without observing what the actual route of technological progress is seems like a deeply scattershot effort.
Conversely, if we work with the two frontier labs to follow processes like “Project Glasswing” and government vetting, we get to see what future models are capable of. We can run objective experiments against reality, instead of building hypothetical mathematical frameworks.
I really don’t think we need a pause for that—maybe just a small speed bump in public capabilities. This also has the advantage of not requiring any sort of international cooperation.
Objection 5 - Dispersal of Power
All of this also means that the longer we pause, the easier it becomes for someone to build something dangerous. It’s easy to monitor and regulate two frontier labs. It’s harder to do that across a dozen labs on three continents. It will become impossible when any of the thousands of companies with large data centers could unleash a rogue ASI.
Even worse, basically all of the Acceleration section amplifies this—everyone will have better computers, better harnesses, and access to a surge of academic research.
If we really did manage a pause, and it didn’t break down ever, it would be good—but I don’t think that’s a realistic scenario compared to the risks of it breaking down, or someone going rogue. And I think a pause actively puts us in a worse situation if things break down early.
A lot of this is specific to our particular timeline—I was really impressed with Project Glasswing, and I think Anthropic is probably one of the best possible stewards of this technology. I want to see someone like Anthropic “win”.
I’ll probably be thinking about this and re-reading it a few times. These are just my initial thoughts after finishing a read of the core “Plan A” + possible failure state sidebars within that.
I don’t think we can pause
My own bias is that if we can pause, we should. I will be quite happy if the future proves me wrong about this. But if we cannot pause, then I desire to believe that we cannot pause. (https://www.lesswrong.com/w/litany-of-tarski)
I do think this is a really invaluable analysis. I think if we are going to pause, our best bet is to rally around something like this.
There’s a huge bipartisan outcry against AI right now, and giving that a concrete rallying point would be really powerful—but that crowd speaks a very different language, and has very different concerns. I think if you really want a pause, you might want to consider pivoting just enough to try and appeal to that crowd—if you can get the artists and writers and displaced programmers behind you, I think this has a much better chance of success.
Personally, my biggest concern with a pause is the risk of dispersal—it’s a lot easier to control two frontier labs, and Anthropic has been an unusually responsible steward compared to most companies and governments.
Objection 1 - Fast Timelines
AI 2027 and Situational Awareness seem to converge on the idea that we could see something like ASI by 2030. That gives us four years. International politics move very slowly—getting a pause in place within two years seems like a political miracle to me, and even three years feels like a stretch. In terms of these timelines, we’re talking about slamming on the brakes. Moving slowly risks missing the window of opportunity.
This is less an objection to Pausing, and more skepticism about the viability
Objection 2 - The Economy
Right now the entire US economy is a massively leveraged bet on AI capabilites continuing to improve. If we slam on the brakes, there’s a good chance the bubble pops, resulting in massive economic turmoil. This means that a lot of people have very good reasons to oppose a pause, and whatever politician implements it might be committing career suicide.
I’d personally be willing to pay this cost, but again, it increases my skepticism about how much buy-in you can actually get.
Objection 3 - Acceleration
The other side of Economy is the profit motive: if some subset of capabilities are blocked, people can still work on making current models cheaper, more efficient, and building an ecosystem of harnesses and tools that let us get more out of existing capabilities. The more you crack down on this, the more you amplify the economic damage of a pause—at the extremes you risk the extinction of the entire industry.
These same motivations apply to academics: people still need to run studies and publish papers to survive, so they’re going to keep doing that. Even if you crack down on publication, a lot of research is still going to be done in private, in anticipation of the pause lifting.
Both of these mean that the instant a pause drops, progress suddenly surges forward. The next model can be trained around a more powerful ecosystem, and incorporate a huge corpus of previously unpublished work. Academics suddenly jumps forward to make up for the missing years.
This is where I start to be concerned about whether a Pause is actually beneficial.
All of this also means that every precaution built during that pause has to stand up to the returned weight of exponential progress *and* a sudden pent-up surge. On top of that, hardware and other more basic capabilities will have advanced, so exponential progress again jumps forward as training runs suddenly skip a step.
Objection 4 - Aligning the Unknown
It’s hard to anticipate what a 10x model will be capable of. It’s a lot easier to solve problems once you know what they are. Trying to build alignment without observing what the actual route of technological progress is seems like a deeply scattershot effort.
Conversely, if we work with the two frontier labs to follow processes like “Project Glasswing” and government vetting, we get to see what future models are capable of. We can run objective experiments against reality, instead of building hypothetical mathematical frameworks.
I really don’t think we need a pause for that—maybe just a small speed bump in public capabilities. This also has the advantage of not requiring any sort of international cooperation.
Objection 5 - Dispersal of Power
All of this also means that the longer we pause, the easier it becomes for someone to build something dangerous. It’s easy to monitor and regulate two frontier labs. It’s harder to do that across a dozen labs on three continents. It will become impossible when any of the thousands of companies with large data centers could unleash a rogue ASI.
Even worse, basically all of the Acceleration section amplifies this—everyone will have better computers, better harnesses, and access to a surge of academic research.
If we really did manage a pause, and it didn’t break down ever, it would be good—but I don’t think that’s a realistic scenario compared to the risks of it breaking down, or someone going rogue. And I think a pause actively puts us in a worse situation if things break down early.
A lot of this is specific to our particular timeline—I was really impressed with Project Glasswing, and I think Anthropic is probably one of the best possible stewards of this technology. I want to see someone like Anthropic “win”.
I’ll probably be thinking about this and re-reading it a few times. These are just my initial thoughts after finishing a read of the core “Plan A” + possible failure state sidebars within that.