OP here. I think this post has, unfortunately for the rest of us, aged quite well. In 2025, OpenAI secured up to $1.5T in compute deals (without much in the way of formal advice), and the industry is collectively investing gargantuan sums to build data centers. While this particular example may seem less dramatic than the others in the “not consistently candid” canon, it’s a very important one.
garrison
Karma: 1,684
Help me launch Obsolete: a book aimed at building a new movement for AI reform
The End of OpenAI’s Nonprofit Era
Anthropic Faces Potentially “Business-Ending” Copyright Lawsuit
Yup, discussed here: https://x.com/GarrisonLovely/status/1926095320997368319?t=vfuPigtomkOn5qc9Z8jCmQ&s=19
I think this is a more significant walkback than what I originally reported. Great find!
The original RSP text uses the word commit for the relevant quote. They could have described it differently in the text itself if they weren’t sure about meeting that standard in the future.
IMO it’s less about the object level badness of the change, which is small potatoes compared to many other recent cases from other leading AI companies, but more about the meta level point that commitments that can be changed aren’t worth very much.
Anthropic is Quietly Backpedalling on its Safety Commitments
Not meaning to imply that Anthropic has dropped ASL-4! Just wanted to call out that this is does represent a change from the Sept. 2023 RSP.
I wrote the article Mikhail referenced and wanted to clarify some things.
The thresholds are specified, but the original commitment says, “We commit to define ASL-4 evaluations before we first train ASL-3 models (i.e. before continuing training beyond when ASL-3 evaluations are triggered). Similarly, we commit to define ASL-5 evaluations before training ASL-4 models, and so forth,” and, regarding ASL-4, “Capabilities and warning sign evaluations defined before training ASL-3 models.”
The latest RSP says this of CBRN-4 Required Safeguards, “We expect this threshold will require the ASL-4 Deployment and Security Standards. We plan to add more information about what those entail in a future update.”
Additionally, AI R&D 4 (confusingly) corresponds to ASL-3 and AI R&D 5 corresponds to ASL-4. This is what the latest RSP says about AI R&D 5 Required Safeguards, “At minimum, the ASL-4 Security Standard (which would protect against model-weight theft by state-level adversaries) is required, although we expect a higher security standard may be required. As with AI R&D-4, we also expect an affirmative case will be required.”
You’re right that contradicted is too strong a word here, though I think OpenAI’s new claim does cast doubt on the earlier reported claim.
I think the fact that investors appear to be fine with this new arrangement is the biggest tell that it’s not a very significant change from the original plan. OpenAI’s nonprofit mission is to ensure AGI benefits humanity, not that it be the first to build it. Legally, the organization has to show that its charitable purpose isn’t changing or that it has a sufficient justification to change it.
There’s more context in my past coverage of the restructuring effort.
What OpenAI Told California’s Attorney General
Four Predictions About OpenAI’s Plans To Retain Nonprofit Control
Sorry, lot on my plate.
You’re basically asking how we’d operationalize the claim that either the USG or PRC are “racing toward AGI”? Probably would involve some dramatic action like consolidating large amounts of compute into projects that are either nationalized or contracted to the govt (like this part of AI-2027:
“A Centralized Development Zone (CDZ) is created at the Tianwan Power Plant (the largest nuclear power plant in the world) to house a new mega-datacenter for DeepCent, along with highly secure living and office spaces to which researchers will eventually relocate. Almost 50% of China’s AI-relevant compute is now working for the DeepCent-led collective,38 and over 80% of new chips are directed to the CDZ.39 At this point, the CDZ has the power capacity in place for what would be the largest centralized cluster in the world.”
Do you want to suggest specific thresholds or modifications?
OpenAI Alums, Nobel Laureates Urge Regulators to Save Company’s Nonprofit Structure
Inside OpenAI’s Controversial Plan to Abandon its Nonprofit Roots
Top OpenAI Catastrophic Risk Official Steps Down Abruptly
I’m hiring a Research Assistant for a nonfiction book on AI!
I don’t think it’s fair to say I made a bad prediction here.
Here’s the full context of my quote: “The report clocks in at a cool 793 pages with 344 endnotes. Despite this length, there are only a handful of mentions of AGI, and all of them are in the sections recommending that the US race to build it.
In other words, there is no evidence in the report to support Helberg’s claim that “China is racing towards AGI.”
Nonetheless, his quote goes unchallenged into the 300-word Reuters story, which will be read far more than the 800-page document. It has the added gravitas of coming from one of the commissioners behind such a gargantuan report.
I’m not asserting that China is definitively NOT rushing to build AGI. But if there were solid evidence behind Helberg’s claim, why didn’t it make it into the report?”
Here’s my tweet mentioning Gwern’s comment. It’s not clear that DeepSeek falsifies what Gwern said here:
the scientific culture of China is ‘mafia’ like (Hsu’s term, not mine) and focused on legible easily-cited incremental research, and is against making any daring research leaps or controversial breakthroughs...
but is capable of extremely high quality world-class followup and large scientific investments given a clear objective target and government marching orders
V3 and R1 are impressive but didn’t advance the absolute capabilities frontier. Maybe the capabilities/cost frontier, though we don’t actually know how compute efficient OAI, Anthropic, GDM are.
I think this part of @gwern’s comment doesn’t hold up as well:
2. there is no interest or investment in an AI arms race, in part because of a “quiet confidence” (ie. apathy/lying-flat) that if anything important happens, fast-follower China can just catch up a few years later and win the real race. They just aren’t doing it. There is no Chinese Manhattan Project. There is no race. They aren’t dumping the money into it, and other things, like chips and Taiwan and demographics, are the big concerns which have the focus from the top of the government, and no one is interested in sticking their necks out for wacky things like ‘spending a billion dollars on a single training run’ without explicit enthusiastic endorsement from the very top.
I still don’t think DS is evidence that “China” is racing toward AGI. The US isn’t racing toward AGI either. Some American companies are, with varying levels of support from the government. But there’s a huge gap between that and Manhattan Project levels of direct govt investment, support, and control.
However, overall, I do think that DS has gotten the CCP more interested in AGI and changed the landscape a lot.
Yeah, realize there’s not as much description as there could be. Basically a new camp in the 3 sided AGI debate (safety, boosters, ethics). Here’s a relevant excerpt:
There are already people who believe that the decisions made about this technology should not be left to those currently making them. Contrary to those who dismiss AI as a “stochastic parrot,” they take AI’s potential seriously. And contrary to classic AI safety, they recognize that there will be no purely technical “solutions” to the problems AI presents. Instead, they see governance as the essential lever. In this book, I’ll call them AI reformers.
Deep learning is incredibly powerful, but it’s being pointed at the wrong things by the wrong people for the wrong reasons. Indeed, its potential actually means it’s even more important that we change how it’s developed. The Obsoleting Project occasionally produces genuine breakthroughs—in medicine, in science—but these are the scraps that fall from the table, not the meal itself. This isn’t an accident; it’s just what markets reward.
Many people on the right and left would like to just press stop on AI. They hate what it’s done to the internet, they hate having it foisted upon them, and they hate the people making it. I get it. Really. (I miss being able to use em-dashes without drawing suspicion, and, sometimes, a sentence structure really isn’t just this, it’s also that!) But this understandable revulsion can blind us to the staggering potential of a reformed AI, where the immense power of deep learning is applied not to produce slop and madness, but to enable scientific discovery and unambiguously better lives—more AlphaFold and less chatbot psychosis.
Crucially, reformers recognize that AI’s present harms and its potential catastrophes aren’t separate problems requiring separate approaches—they’re symptoms of the same underlying dynamics: competitive pressure, concentrated power, and a staggering lack of accountability. The chatbot that encourages a teenager’s suicide and the hypothetical superintelligence that slips human control both emerge from organizations racing to deploy systems they don’t really understand. Getting serious about one means getting serious about the other. Reformers recognize that, if it were aimed in different directions, the technology could be so much better than it is. They realize that the people advancing the Obsoleting Project—not the various people looking to change or stop that project—are the ones with the real power, at least for now. They believe that a better world is possible, and so is better AI. In short, AI reform works toward the version of AI that the Obsoleting Project sells, but doesn’t actually make.