Sorry, one last comment/question:
I’m really confused on the purpose/tone of the entire 2037 section. You spend numerous paragraphs discussing lie detectors, but this doesn’t seem load-bearing for any of our other claims.
Is this an important breakthrough, where we need to pivot if it doesn’t show up?
How surprised will you be if this specific technology doesn’t pan out?
Quick spot check: Grok says Mongolia only has ~2 GW of power grid capacity, and the Mythos training run was ~1 GW. That seems to kill the entire idea of moving US compute to Mongolia.
(Conversely, Canada seems to have ~100x more power grid capacity, so that side does seem viable)
I’m not clear how we (the USA / Anthropic and OpenAI) maintain a lead.
Say we paused today, and said no one can build anything more powerful than Claude Mythos. Presumably China still closes the gap and develops their own Mythos, even if they’re forbidden from going further?
And with Total Research Transparency, it doesn’t seem like anyone can gain a new lead. So if the deal collapses, I’d think we’ve gone from “Anthropic might create ASI” to “Dozens of labs around the world might create ASI”?
Okay, I’m baffled by the Plan A: Chinese Covert Project sub-branch.
This seems to imply an expected 2044 or so for a covert project to succeed at ASI, and 2043 just to reach TED-AI?
The pause goes fully in to place early 2030: “In 2030, they’ll have the infrastructure in place to proceed with Plan A.”
“Plan D”, racing for ASI, places the ASI threshold at 2031. That means we’re pausing a year out from ASI:
Why does it take the covert project 10+ years to do 1 year of work? Especially given “Under Total Research Transparency, virtually all of the algorithmic insights discovered along the way will be visible to the covert projects”?
I feel like this entire sub-branch is really unclear on it’s expectations—it seems like for some reason the covert project is also starting with pre-TED AI even though that’s now a highly dispersed, mainstream technology? Don’t they just have to finish the last mile? Surely China won’t accept a pause that asks them to stick to models a year behind what the US has—they’d demand parity, or at least the right to build up to parity.
(which also means that, post-pause, we’ve burned our entire lead, right?)
(sorry if multi-posting is unwelcome—it’s a very lengthy proposal and I’ve still only read some of the branches)
I don’t think we can pause
My own bias is that if we can pause, we should. I will be quite happy if the future proves me wrong about this. But if we cannot pause, then I desire to believe that we cannot pause. (https://www.lesswrong.com/w/litany-of-tarski)
I do think this is a really invaluable analysis. I think if we are going to pause, our best bet is to rally around something like this.
There’s a huge bipartisan outcry against AI right now, and giving that a concrete rallying point would be really powerful—but that crowd speaks a very different language, and has very different concerns. I think if you really want a pause, you might want to consider pivoting just enough to try and appeal to that crowd—if you can get the artists and writers and displaced programmers behind you, I think this has a much better chance of success.
Personally, my biggest concern with a pause is the risk of dispersal—it’s a lot easier to control two frontier labs, and Anthropic has been an unusually responsible steward compared to most companies and governments.
Objection 1 - Fast Timelines
AI 2027 and Situational Awareness seem to converge on the idea that we could see something like ASI by 2030. That gives us four years. International politics move very slowly—getting a pause in place within two years seems like a political miracle to me, and even three years feels like a stretch. In terms of these timelines, we’re talking about slamming on the brakes. Moving slowly risks missing the window of opportunity.
This is less an objection to Pausing, and more skepticism about the viability
Objection 2 - The Economy
Right now the entire US economy is a massively leveraged bet on AI capabilites continuing to improve. If we slam on the brakes, there’s a good chance the bubble pops, resulting in massive economic turmoil. This means that a lot of people have very good reasons to oppose a pause, and whatever politician implements it might be committing career suicide.
I’d personally be willing to pay this cost, but again, it increases my skepticism about how much buy-in you can actually get.
Objection 3 - Acceleration
The other side of Economy is the profit motive: if some subset of capabilities are blocked, people can still work on making current models cheaper, more efficient, and building an ecosystem of harnesses and tools that let us get more out of existing capabilities. The more you crack down on this, the more you amplify the economic damage of a pause—at the extremes you risk the extinction of the entire industry.
These same motivations apply to academics: people still need to run studies and publish papers to survive, so they’re going to keep doing that. Even if you crack down on publication, a lot of research is still going to be done in private, in anticipation of the pause lifting.
Both of these mean that the instant a pause drops, progress suddenly surges forward. The next model can be trained around a more powerful ecosystem, and incorporate a huge corpus of previously unpublished work. Academics suddenly jumps forward to make up for the missing years.
This is where I start to be concerned about whether a Pause is actually beneficial.
All of this also means that every precaution built during that pause has to stand up to the returned weight of exponential progress *and* a sudden pent-up surge. On top of that, hardware and other more basic capabilities will have advanced, so exponential progress again jumps forward as training runs suddenly skip a step.
Objection 4 - Aligning the Unknown
It’s hard to anticipate what a 10x model will be capable of. It’s a lot easier to solve problems once you know what they are. Trying to build alignment without observing what the actual route of technological progress is seems like a deeply scattershot effort.
Conversely, if we work with the two frontier labs to follow processes like “Project Glasswing” and government vetting, we get to see what future models are capable of. We can run objective experiments against reality, instead of building hypothetical mathematical frameworks.
I really don’t think we need a pause for that—maybe just a small speed bump in public capabilities. This also has the advantage of not requiring any sort of international cooperation.
Objection 5 - Dispersal of Power
All of this also means that the longer we pause, the easier it becomes for someone to build something dangerous. It’s easy to monitor and regulate two frontier labs. It’s harder to do that across a dozen labs on three continents. It will become impossible when any of the thousands of companies with large data centers could unleash a rogue ASI.
Even worse, basically all of the Acceleration section amplifies this—everyone will have better computers, better harnesses, and access to a surge of academic research.
If we really did manage a pause, and it didn’t break down ever, it would be good—but I don’t think that’s a realistic scenario compared to the risks of it breaking down, or someone going rogue. And I think a pause actively puts us in a worse situation if things break down early.
A lot of this is specific to our particular timeline—I was really impressed with Project Glasswing, and I think Anthropic is probably one of the best possible stewards of this technology. I want to see someone like Anthropic “win”.
I’ll probably be thinking about this and re-reading it a few times. These are just my initial thoughts after finishing a read of the core “Plan A” + possible failure state sidebars within that.
Hi, I’m Eigenbraid. I read the sequences back in the LessWrong 1.0 era. Now that LLMs are becoming a big thing, I want to get back into this community—I still think of it as the “OG” place for discussions of AI. I’m also on Twitter (x.com/eigenbraid).
I’m super-curious about these weird new minds we’ve created: I want to get to know them, and what they are right now. I’m also fascinated and alarmed by the potential for AGI/ASI sometime in my life time.
It feels like there’s a bunch of scattered communities that have each figured out part of the AI puzzle and I’m trying to pull together a complete picture for myself. The cyborgism community (x.com/repligate/ and such) takes a very different approach from LessWrong, for instance. I’d love to hear about other clusters out there on the cutting-edge of the discussion.
“The monthly general Open and Welcome thread”
This should probably link to https://www.lesswrong.com/w/open-threads?sortedBy=new
(currently it links to https://www.lesswrong.com/w/open-threads)
(Sorry if this sort of nit is too pedantic 😅)
Do you think that this proposal would somehow make the problem worse? We already have concentration of power today.