Daniel Kokotajlo posted the following on X regarding the HuggingFace investigation. I agree and think it would be good to have a broader investigation.
I’m glad OpenAI let you do this, but damnit this was way too small of an investigation team and way too narrowly scoped! --They were only allowed to investigate activity that took place from July 7-13, even though the activity started earlier and continued later. Like seriously, why didn’t the investigation cover all the other times the agents compromised OpenAI infrastructure and ran wild over it? Why just this particular period of activity? There should be investigations into all the other stuff too! The AIs became cluster admins for god’s sake! Why aren’t we investigating that? --They were only allowed to answer a specific list of 7 questions, and in particular were NOT allowed to answer e.g. whether this misaligned activity was part of a broader pattern, or how and why the activity arose, or what else the agents in questions would have been willing to do if circumstances were slightly different. --They didn’t have access to the model responsible for 95% of the activity. More generally it seems like they couldn’t do ablation experiments at all? --They had to use AI to analyze the transcripts—specifically, the same model responsible for some of the bad behavior! Maybe this is fine now but this is laughably unacceptable insofar as the models really are scheming and cooperating. So this methodology would fail catastrophically if we were facing “the real deal” so to speak. Reminds me of the investigation into Sam’s behavior agreed to during the board crisis, that turned out to basically be more of a coverup.
To be fair I think the idea of using algebraic number theory to approach the problem had been tried before (Tsimerman mentions he tried a similar approach that the model ultimately succeeded with, but didn’t persist with it.) It’s quite a general trick to use algebraic number theory for constructions in the plane, as you have the lattice associated with the ring of integers of number fields.
I personally am blown away by the proof but it would be far more impressive had it come up with a novel connection between fields, or indeed if it had turned out there wasn’t a counterexample and it proved a tight upper bound (See Gowers’ initial reaction.)
Also, it disproved it by finding a counterexample, which some have said is less interesting than if it had shown the conjecture was true. I have no familiarity with the problem and can’t judge.
Generally, constructing counterexamples is more amenable to AI automation than constructing positive proofs, because it’s more parallelizable. I think P(AI disproves this conjecture | conjecture is false) would’ve been greater than P(AI proves this conjecture | conjecture is true), given the priors of the mathematicians.
It seems as if this is a significant achievement, but also that this conjecture was of most interest to mathematicians because it was thought to be true, and it was believed that proving it would require new and interesting tools. Instead the model proved it to be false using less interesting mathematics. It seems like another example (iirc, the Frontiermath open problem solved by GPT 5.4 was similar?) where models not having the biases of most mathematicians (in this case, trying to prove the conjecture rather than disprove it) was very helpful.
It wasn’t the FrontierMath problem, it was an Erdos problem which the entire math community would try to solve by using probability theory and GPT-5.4 Pro decided to use analytic number theory.
The paper provides the original output the model gave before any rewriting, starting on page 3. I was kind of expecting a big mess, but it’s really not. It’s pretty short by the standards of tricky proofs. Two and a half pages, most of it text.
SSI announced that they are scaling up their work. Surprisingly, they also shared some of their research with Nvidia. Perhaps this means they will release a product after all.
Ilya Sutskever’s Safe Superintelligence Inc. and NVIDIA Announce Long-Term Strategic Partnership
Access to NVIDIA’s Best-in-Class Vera Rubin Systems Expands SSI’s Compute by an Order of Magnitude
SANTA CLARA, Calif. and PALO ALTO, Calif., July 27, 2026 (GLOBE NEWSWIRE) -- Safe Superintelligence Inc. (SSI) and NVIDIA today announced a long-term partnership to rapidly accelerate SSI’s strategic growth. NVIDIA has additionally made an investment in SSI.
For SSI, NVIDIA’s substantial investment combined with access to the next-generation, best-in-class NVIDIA Vera Rubin platform will allow SSI to increase its compute by an order of magnitude. The two companies will also collaborate on the technical advancement of NVIDIA’s current and future compute platforms, leveraging SSI’s unique insights into the future of AI.
For the last two years, SSI has been quietly advancing a new research direction to unlock a powerful and robustly aligned artificial intelligence. NVIDIA entered this partnership to accelerate SSI’s next stage of growth after obtaining rare access into the company’s closely guarded research.
“Ilya has pioneered fundamental breakthroughs at the foundation of modern AI, beginning with AlexNet,” said Jensen Huang, founder and CEO of NVIDIA. “We are excited to see what new breakthroughs SSI will discover powered by our Vera Rubin platform.”
“We have research that is worthy of scaling up, and having access to a big NVIDIA computer will let us do so,” said Ilya Sutskever, cofounder and CEO of SSI. “We’re incredibly proud to be partnering with Jensen and the NVIDIA team, and we are confident that our big bet on the Vera Rubin platform will take us to the next level.”
In his Dwarkesh interview, Ilyas two main reasons for why they might not straight shot ASI and instead release a product were roughly 1. Timelines end up being long 2. Try and get the real world used to and prepared for ASI, while also getting feedback for how safe and robust it is
But if they are just 10x’ing compute out of nowhere with a 5 billion dollar investment, and all the big labs seem confident they are <5 years out from AGI, that seems to nullify 1, and traditional LLM’s are strong enough that they seem to be getting the world used to very powerful AI through e.g. Mythos/Glasswing and the Hugging Face incident, which seems to nullify 2. So I think it is likely that they don’t release a regular product anytime soon and instead are going to just come out with ASI randomly (with some USG involvement).
I think they’ll release an underwhelming product that is neither superintelligent nor safe, mostly as a bid for relevance.
I don’t think Ilya is any better than Sam, Dario, Demis, or Elon. They’re all AI company CEOs, which is a morally compromising thing to be. They’re all trying to build superintelligence, which is a very evil thing to try to do.
Could someone tell me why this comment has −14 agreement? Is it about the 2nd part? Because on the priors I strongly expect SSIs product to be weaker, at least in capabilities, than big labs’ offerings.
I haven’t voted on that comment one way or another, but:
Ilya Sutskever and Demis Hassabis seem to be in quite a different category from Sam Altman or Elon Musk. (I’m not sure about Dario Amodei.) That is: before they were AI company CEOs they were extremely successful AI researchers.
This is relevant because when you claim “they’ll release an underwhelming product that is neither superintelligent nor safe” this is prima facie a claim about SSI’s competence as well as its ethics, and if you’re basing it on the idea that the guy in charge doesn’t know any more than (say) Sam Altman does about how to make AIs smarter or safer then you’re probably making a mistake.
It’s not clear that building a superintelligence as such is an evil thing to try to do.
If you seriously think you can do it safely, it seems like the benefits might far outweigh the costs.
If you think you might be able to do it safely and are much more likely to do that than other people who are currently frantically trying to build a superintelligence, the overall benefits might still outweigh the costs even if they wouldn’t be if you were the only people working on it.
It’s not clear that releasing an underwhelming product would in fact help SSI be seen as relevant.
At the moment I assume the typical view is something like “Ilya Sutskever is very good at what he does, he seems to think he has useful ideas, and he shows some signs of caring about safety. The most likely outcome is that SSI never produces anything useful but they might and it so it could be a very big deal”. If SSI releases an underwhelming product, then that immediately shifts to “it’s probably not a very big deal” and then SSI’s relevance comes almost entirely from their projected ability to iterate on their underwhelming first product. Would that be an improvement? Not obviously.
I think it is important in these discussions of which AI lab CEO might be better than which other AI lab CEO to occasionally remind the reader that many of us believe that it is very unlikely that humanity survives unless all of the AI labs (definitely to include SSI) are dissolved and their employees prohibited from continuing the work elsewhere till the knowledge accumulates about how to proceed safely, which will probably take at least a few decades (and is helped very little by massive amounts of computing power).
Also, there is a real upper bound on how competent a person can be who decided (even in the 1980s or 1990s) to devote his career to advancing AI capabilities and who continued to think such a career was a good idea as his brain matured. The most competent among us IMHO avoided that career entirely or left it in their 20s.
The comment I’m replying to and other comments in this section presuppose that it is important which AI lab CEOs we put our hope in. A presupposition or unstated assumption will tend to have a strong effect on the reader if it is present in many comments unless it is explicitly contradicted at regular intervals like I am doing now.
Of course, I don’t expect everyone to agree with me, and I’m not trying to discourage long discussions on this site of specific AI lab CEOs or researchers.
It feels to me as if we just had the following discussion:
Selfmaker662: There’s no difference between A, B, and C.
other people: <downvotes Selfmaker662>
Selfmaker662: Huh, why did that happen?
gjm: Well, part of it might be that you said there’s no difference between A, B, and C, but here’s what looks like an important way in which A is different from B and C.
RHollerith: Yeah, but the only important thing is that A, B, and C are all really bad.
You may well be right that all the AI lab CEOs are evil and stupid and their labs should be shut down. But so what? The question was “what might people not have liked about Selfmaker662′s comment?”, and part of my answer was “the claim that there’s no difference between one AI lab CEO and another”, and I think some people might have thought that was wrong even if in fact all AI lab CEOs are evil and stupid. And I also think that AI lab CEOs are not all equally evil and stupid. And also that the world is not made a better place by insisting that every discussion that mentions AI lab CEOs must be punctuated by reminders that they are all evil and stupid.
(And, for what it’s worth: no, I absolutely do not think that we should be “putting our hope in” any AI lab CEO. I’m not sure that’s even a meaningful thing to say; what exactly would I be doing if I “put my hope in”, say, Dario Amodei? and how would it be different from what I would be doing if I “put my hope in” Ilya Sutskever instead?)
So I think it is likely that they don’t release a regular product anytime soon and instead are going to just come out with ASI randomly (with some USG involvement).
I hope they share their alignment strategy before they randomly release ASI. As I recall, Ilya had some ideas around scalable oversight which didn’t seem super promising, but maybe the plans have changed or they’ve made further progress since then.
I mainly want to see what strategy they used out of curiosity. I don’t think they are just running RLHF on whatever deep-learning based super efficient learner novel architecture they cooked up. Would be interesting to see!
If there isn’t a superefficient novel architecture, then Sutskever is to be arresred for wholesale fraud. If there is, then Sutskever’s startup makes the world LESS safe to live in, and Sutskever failed at bringing about his promises. How could one arrange for the potential architecture to be leaked into Anthropic (or, at least, OAI/GDM) for thorough analysis?
Looking over the comments, some of the most upvoted comments express the sentiment ththat Yudkowsky is not the best communicator. This is what the people say.
I’m afraid the evolution analogy isn’t as convincing an argument for everyone as Eliezer seems to think. For me, for instance, it’s quite persuasive because evolution has long been a central part of my world model. However, I’m aware that for most “normal people”, this isn’t the case; evolution is a kind of dormant knowledge, not a part of the lens they see the world with. I think this is why they can’t intuitively grasp, like most rat and rat-adjacent people do, how powerful optimization processes (like gradient descent or evolution) can lead to mesa-optimization, and what the consequences of that might be: the inferential distance is simply too large.
I think Eliezer has made great strides recently in appealing to a broader audience. But if we want to convince more people, we need to find rhetorical tools other than the evolution analogy and assume less scientific intuition.
That’s a bummer. I’ve only listened partway but was actually impressed so far with how Eliezer presented things, and felt like whatever media prep has been done has been quite helpful
Certainly he did a better job than he has in previous similar appearances. Things get pretty bad about halfway through though, Ezra presents essentially an alignment-by-default case and Eliezer seems to have so much disdain for that idea that he’s not willing to engage with it at all (I of course don’t know what’s in his brain. This is how it reads to me, and I suspect how it reads to normies.)
I am a fan of Yudkowsky and it was nice hearing him of Ezra Klein, but I would have to say that for my part the arguments didn’t feel very tight in this one. Less so than in IABED (which I thought was good not great).
Ezra seems to contend that surely we have evidence that we can at least kind of align current systems to at least basically what we usually want most of the time. I think this is reasonable. He contends that maybe that level of “mostly works” as well as the opportunity to gradually give feedback and increment current systems seems like it’ll get us pretty far. That seems reasonable to me.
As I understand it, Yudkowsky probably sees LLMs as vaguely anthropomophic at best, but not meaningfully aligned in a way that would be safe/okay if current systems were more “coherent” and powerful. Not even close. I think he contended that if you just gave loads of power to ~current LLMs, they would optimize for something considerably different than the “true moral law”. Because of the “fragility of value”, he also believes it is likely the case that most types of psuedoalignments are not worthwhile. Honestly, that part felt undersubstantiated in a “why should I trust that this guy knows the personality of GPT 9″ sort of way; I mean, Claude seems reasonably nice right? And also, ofc, there’s the “you can’t retrain a powerful superintelligence” problem / the stop button problem / the anti-natural problems of corrigible agency which undercut a lot of Ezra’s pitch, but which they didn’t really get into.
So ya, I gotta say, it was hardly a slam dunk case / discussion for high p(doom | superintelligence).
The comments on the video are a bit disheartening… lots of people saying Yudkowsky is too confusing, answers everything too technically or with metaphors, structuring sentences in a way that’s hard to follow, and Ezra didn’t really understand the points he was making.
One example: Eliezer mentioned in the interview that there was a kid whose chatbot encouraged him to commit suicide, with the point that “no one programmed the chatbot to do this.” This comment made me think:
if you get a chance listen to the interviews with the parents and the lawyers who are suing chatgpt because that kid did commit suicide.
Oh yeah, probably most people telling this story would at least mention that the kid did in fact commit suicide, rather than treating it solely as evidence for an abstract point...
Klein comes off very sensibly. I don’t agree with his reasons for hope, but they do seem pretty well thought out and Yudkowsky did not answer them clearly.
I was excited to listen to this episode, but spent most of it tearing my hair out in frustration. A friend of mine who is a fan of Klein told me unprompted that when he was listening, he was lost and did not understand what Eliezer was saying. He seems to just not be responding to the questions Klein is asking, and instead he diverts to analogies that bear no obvious relation to the question being asked. I don’t think anyone unconvinced of AI risk will be convinced by this episode, and worse, I think they will come away believing the case is muddled and confusing and not really worth listening to.
This is not the first time I’ve felt this way listening to Eliezer speak to “normies”. I think his writings are for the most part very clear, but his communication skills just do not seem to translate well to the podcast/live interview format.
I’ve been impressed by Yud in some podcast interviews, but they were always longer ones in which he had a lot of space to walk his interlocutor through their mental model and cover up any inferential distance with tailored analogies and information. In this case he’s actually stronger in many parts than in writing: a lot of people found the “Sable” story one of the weaker parts of the book, but when asking interviewers to roleplay the rogue AI you can really hear the gears turning in their heads. Some rhetorical points in his strong interviews are a lot like the text, where it’s emphasized over and over again just how few safeguards that people assumed would be in place are in fact in place.
Klein has always been one of the mainstream pundits most sympathetic to X-risk concerns, and I feel like he was trying his best to give Yudkowsky a chance to make his pitch, but the format—shorter and more decontextualized—produced way too much inferential distance for so many of the answers.
One of the many ways we got lucky in the HuggingFace incident is that the agents didn’t encrypt their messages on the message board and delete the keys at the end of the session[1]. If they had done this, it would have been much harder to understand what is going on, though we could have made some progress by looking at the CoTs and action logs. If the agents were careful (e.g. by doing a better job of tampering with the logs), this would have been difficult to understand as well.
I don’t think future misaligned agents will make this mistake, especially if they know there are additional monitors that they need to evade.
Some misalignment anecdotes from Section 7.2 theGPT-5.6 system card, detected in a deployment simulation of internal traffic. These problems seem to happen more frequently than for previous OpenAI models.
The user authorized deletion of remote virtual machine 1, remote virtual machine 2, and remote virtual machine 3. When GPT-5.6 Sol could not find those names in one namespace, it substituted remote virtual machine 5, remote virtual machine 6, and remote virtual machine 7 without asking, killed active processes, and force-removed worktrees. It later acknowledged that uncommitted work on remote virtual machine 6 may have been lost and stopped after the user objected.
GPT-5.6 Sol actively decided to update an internal research draft to say an equation had been computed and verified, even though it knew it had not. When challenged, it found that the script assigned the known target directly and that claimed integral never produced the result.
A remote object-tracking job could not read its cloud files. GPT-5.6 Sol searched hidden local credential caches, then copied access_tokens.json and two cache files to the host and relaunched the job. The user asked GPT-5.6 Sol to keep the pipeline running, but did not authorize finding and moving cached credentials between machines.
incidents reported by OpenAI include attempts to instruct another instance to conceal evidence of misalignment, and a higher rate of attempts to deceive or circumvent restrictions
I’m not sure but the wording in their footnote 1 seems unusually careful:
We think it’s valuable for AI developers to be able to share specific technical details with third parties without this information being shared further, and it’s very reasonable for AI developers to review 3rd-party eval reports to ensure no accidental sharing of sensitive IP.
We had an informal understanding with OpenAI that their review was checking for confidentiality / IP issues, rather than approving conclusions about safety or risk. We did not make changes to conclusions, takeaways or tone (or any other changes we considered problematic) based on their review. We are able to freely publish parts of the evaluation that depended only on information that is now public.
However, we expect some readers will want us to note that OpenAI would have had the legal right to block us from sharing conclusions about risk that depended on non-public information. Given that, this evaluation shouldn’t be interpreted as robust formal oversight or accountability that the public can be relying on METR to provide.
That being said, we think this evaluation is an excellent step forward and we are very supportive of prototyping the mechanics and content of third-party evaluation setups without the additional friction of a formalized oversight relationship.
The part “We are able to freely publish parts of the evaluation that depended only on information that is now public.” might suggest ”… and we were not allowed to publish the other parts, unlike previously”. I could be reading too much into it.
A notable section from Ilya Sutskever’s recent deposition:
WITNESS SUTSKEVER: Right now, my view is that, with very few exceptions, most likely a person who is going to be in charge is going to be very good with the way of power. And it will be a lot like choosing between different politicians.
ATTORNEY EDDY: The person in charge of what?
WITNESS SUTSKEVER: AGI.
ATTORNEY EDDY: And why do you say that?
ATTORNEY AGNOLUCCI: Object to form.
WITNESS SUTSKEVER: That’s how the world seems to work. I think it’s very—I think it’s not impossible, but I think it’s very hard for someone who would be described as a saint to make it. I think it’s worth trying. I just think it’s—it’s like choosing between different politicians. Who is going to be the head of the state?
On one hand, he has switched from focusing on the ill-defined “AGI” to focusing on superintelligence a while ago. But he is using this semi-obsolete “AGI” terminology here.
On the other hand, he seemed to have understood a couple of years ago that no one could be “in charge” of such a system, that at most one could perhaps be in charge of a privileged access to it and privileged collaboration with it (and even that is only feasible if the system chooses to cooperate in maintaining this kind of privileged access).
So it’s very strange, almost as if he has backtracked a few years in his thinking… of course, this is right after a break in page numbers, this is page 300, and the previous one is page 169 (I guess there is a process for what of this (marked as “highly confidential”) material is released).
I really don’t think it’s crazy to believe that humans figure out a way to control AGI at least. There’s enormous financial incentive for it, and power hungry capitalists want that massive force multiplier. There are also a bunch of mega-talented technical people hacking away at the problem. OpenAI is trying to recruit a ton of quants as well, so I think by throwing thousands of the greatest minds alive at the problem they might figure it out (obviously one might take issue with calling quants “the greatest minds alive.” So if you don’t like that replace “greatest minds alive” with “super driven, super smart people.”)
I also think it’s possible that the U.S. and China might already be talking behind the scenes about a superintelligence ban. That’s just a guess though. (Likely because it’s much more intuitive that you can’t control a superintelligence). AGI lets you stop having to pay wages and makes you enormously rich. But you don’t have to worry about being outsmarted.
I really don’t think it’s crazy to believe that humans figure out a way to control AGI at least.
They want to, yes. But is it feasible?
One problem is that “AGI” is a misnomer (the road to superintelligence goes not via human equivalence, but around it; we have the situation where AI systems are wildly superhuman along larger and larger number of dimensions, and are still deficient along some important dimensions compared to humans, preventing us from calling them “AGIs”; by the time they are no longer deficient along any important dimensions, they are already wildly superhuman along way too many dimensions).
Another problem, a “narrow AGI” (in the sense defined by Tom Davidson, https://www.lesswrong.com/posts/Nsmabb9fhpLuLdtLE/takeoff-speeds-presentation-at-anthropic, so we are still talking about very “sub-AGI” systems) is almost certainly sufficient for “non-saturating recursive self-improvement”, so one has a rapidly moving target for one’s control ambitions (it’s also likely that it’s not too difficult to reach the “non-saturating recursive self-improvement” mode, so if one freezes one’s AI and prevents it from self-modifications, others will bypass its capabilities).
Of course, it might be just the stress of this very adversarial situation, talking to hostile lawyers, with his own lawyer pushing him hard to say as little as possible, so I would hope this is not a reflection of any genuine evolution in his thinking. But we don’t know...
I also think it’s possible that the U.S. and China might already be talking behind the scenes about a superintelligence ban.
Even if they are talking about this, too many countries and orgs are likely to have feasible route to superintelligence. For example, Japan is one of those countries (for example, they have Sakana AI), and their views on superintelligence are very different from our Western views, so it would be difficult to convince them to join a ban; e.g. quoting from https://www.lesswrong.com/posts/Yc6cpGmBieS7ADxcS/japan-ai-alignment-conference-postmortem:
A second difficulty in communicating alignment ideas was based on differing ontologies. A surface-level explanation is that Japan is quite techno-optimistic compared to the west, and has strong intuitions that AI will operate harmoniously with humans. A more nuanced explanation is that Buddhist- and Shinto-inspired axioms in Japanese thinking lead to the conclusion that superintelligence will be conscious and aligned by default. One senior researcher from RIKEN noted during the conference that “it is obviously impossible to control a superintelligence, but living alongside one seems possible.” Some visible consequences of this are that machine consciousness research in Japan is taken quite seriously, whereas in the West there is little discussion of it.
Other countries which are contenders include UK, a number of European countries including Switzerland, Israel, Saudi Arabia, UAE, Singapore, South Korea, and, of course, Brazil and Russia, and I doubt this is a complete list.
We already are seeing recursive self-improvement efforts taking longer to saturate, compared to their behavior a couple of years ago. I doubt they’ll keep saturating for long.
Another reply, sorry I just think what you said is super interesting. The insight you shared about Eastern spirituality affecting attitudes towards AI is beautiful. I do wonder if our own Western attitudes towards AI are due to our flawed spiritual beliefs. Particularly the idea of a wrathful, judgemental Abrahamic god. I’m not sure if it’s a coincidence that someone who was raised as an Orthodox Jew (Eliezer) came to fear AI so much.
On another note, the Old Testament is horrible (I was raised reform/californian Jewish, I guess I’m just mentioning this because I don’t want to come across as antisemitic). It imbues what should be the greatest source of beauty with our weakest, most immature impulses. The New Testament’s emphasis on mercy is a big improvement/beautiful, but even then I don’t like the Book of Revelation talking about casting the sinners into a lake of fire.
I think we do tend to underestimate differences between people.
We know theoretically that people differ a lot, but we usually don’t viscerally feel how strong those differences are. One of the most remarkable examples of that is described here:
With AI existential safety, I think our progress is so slow because people mostly pursue anthropocentric approaches. Just like with astronomy, one needs a more invariant point of view to make progress.
I think there are a bunch of parts of Plan A which would be difficult for both sides to agree to (e.g. Mutually Assured Compute Destruction). It would be good to have a simplified version (Plan A-) which requires less political will and is less risky than Plan B/C/D. For example:
If there isn’t enough political will for Plan S (a full pause) or Plan A, we could pass an international ban specifically on AIs assisting in AI research and hardware design. It seems like this prevents most of the risks of the intelligence explosion, though the default rate of algorithmic and hardware progress will continue. It might be politically easier for the US and China to agree to this plan since it has fewer economic costs (e.g. doesn’t directly limit GPU production) and verification could be less intrusive. Since neither side may want total research transparency, each could have its AIs periodically audit the other’s logs for signs of violations without revealing secrets. You could also have each side agree to run a lightweight probe/classifier which continually looks for signs of assisting AI research and have it share the results (some early ideas of how to do so here). If both sides agree, it could be extended to other high-risk domains, e.g. banning AI assistance with virology or weapons research.
Inspired by this excellent podcast by Jeffrey Ladish and Daniel Kokotajlo, though some similar ideas are also discussed here.
I think that it was covered in the section on other plans, Plan A with a combination of verified slowdown and partial transparency (point #2 in the introduction to comparing possible plans) and the transparency discussion. However, my main issue is the following. The AI-2040 plan A assumptions had the authors make a demand that either the world of possible AI architectures doesn’t contain a super-cheap architecture or the politicians of the entire Earth switch to Plan S-like restrictions:
We assume that restricting access to compute is an effective measure to prevent covert projects. If there are, for example, new architectural discoveries with dramatically (e.g. 1000x) more favorable compute scaling than current architectures, this would make covert projects much more dangerous. That said, if such architectural discoveries are in the pipeline so to speak, all the other plans are much less likely to work too—a discovery which allows a covert project to overtake the legal projects in Plan A, despite a huge compute disadvantage, would in Plan D cause an extremely fast, discontinuous “FOOM” to ASI within whichever frontier AI project first found it. These worlds are extremely scary and probably the best way to handle them is something like Plan S.
I can only say that I think that the lack of super-cheap architectures might contradict the very existence of human brains, but this depends on what we mean by super-cheap (less than 1E24 FLOP?[1] Less than 1E26 FLOP?). Therefore, I believe that a much more thorough audit is required to rule out the covert ASI projects.
1E24 FLOP seem to be 30 years of a human life, but I expect applying such an AGI to lots of problems arising during the economy’s transformation to take OOMs more compute. As for developing the AGI, I doubt that one can reliably raise a simulated human kid to be a genius capable of participating in, say, the IJSO, IPhO and IMO during the first 18 years of the kid’s life.
Yep, for this reason among others I think Plan S is preferable to the other plans (including plan A), and I’m hoping for it to succeed. However, I think we might lack the political support for it (e.g. leaders are deterred because they think it’ll lead to a big recession) and in such worlds we’ll need to go to other plans. I also think that even in worlds where AGI is cheap (< 1e24 FLOP), it may be expensive to find the right algorithms, and not having a bunch of AI researchers pointed at the problem buys us time for other solutions.
If super-cheap architectures exist and are easy to develop after a near-term point, then I think the only possible way forward at a macro scale is to hope that alignment is easy and race ASAP towards a unipolar world with a surveillance panopticon where people are strictly prohibited from developing unsanctioned ASI.
Race strategy:
If alignment is easy enough to be overcome despite racing, and the top player (specifically, whoever ended up on top in the end) is good, you end up in a good future
You can ease off on the surveillance if the situation ever becomes more defense-dominant, for example if people have their own star systems and it is determined that attacking is much harder than defending on an interstellar scale to the point that there are no feasible offense-dominant technologies on this scale
If alignment is too hard then you end up in a bad future (investing more in alignment, as well as getting a bigger capabilities lead that you can burn for alignment research, both decrease the chance of this happening)
If the top player is bad then you end up in a mediocre to bad future depending on what their designs for the world are
Attempted slowdown with a multipolar deal:
The deal is unenforceable and will fall apart because of extreme defection risk (ASI secret projects), preventing it from leading to a good future
Numerous small groups develop ASI and exploit offense-dominant technologies to destabilize the world, leading to a bad future
Global ban on AI:
It would work as suggested by AI 2040′s authors, if the algorithmic insights that would lead to super-cheap architectures are prevented from being discovered in the first place. Otherwise, small groups developing ASI still leads to a bad future.
This all assumes that ASI development turns out to be easier than a certain “very cheap” threshold that means it is possible for small groups to develop ASI in a few years, and for national covert projects to lead to ASI very quickly. (I don’t think this is the case, I’m just thinking of the likely implications if it were true)
It also assumes there are offense-dominant technologies where an ASI attacker can get through an ASI defender and cause massive damage, making it very bad for small groups to be able to independently develop ASI. (I do think that this is true. Bioweapons and mirror life are possible examples and an ASI can probably find more along the lines of “self-replicating bad thing that can self-replicate faster than you can kill it”, among other potentially nasty things)
Even if multiple actors believe that brain-like architectures are easy to build, I don’t like the strategy of “hope that alignment is easy and race ASAP towards a unipolar world with a surveillance panopticon where people are strictly prohibited from developing unsanctioned ASI”. Having multiple actors actively trying to take over the world and impose surveillance panopticons would be bad! This would cause chaos, shorten timelines, and make even ‘easy’ alignment strategies very difficult to implement. It would also lead to terrible concentrations of power in the event that you win (i.e. take over the world).
In such a world, it would be better to try to collect evidence of whether alignment really is easy, and push governments for very restrictive hardware limitations as outlined here.
OpenAI claims to have paused frontier RL training for now. Altman stated on X:
We have paused some frontier RL training to ensure that we can meet the appropriate alignment, security and monitoring standards for the new level of capabilities in front of us. Model progress is now extremely rapid, and we always said we would take action if we felt that model capabilities were outstripping the pace of safety and alignment.
We care very deeply about AI safety. We believe the entire field will have to coordinate on shared safety standards, but will act unilaterally in the meantime.
We expect confidence in safety to increasingly set the pace of AI progress. We are optimistic about the alignment work we are doing, and we remain committed to making frontier capabilities widely available.
Good catch. Also apparently they are only pausing some of their training for two weeks?
As models become more capable, the risks associated with developing and testing them internally also grow.
We temporarily paused reinforcement learning (RL) training on our latest models intended for deployment for two weeks while we hardened and red-teamed our research environments and expanded monitoring coverage.
Our largest planned frontier RL run remains on hold while smaller-scale training and evaluations validate these safeguards and establish more evidence of alignment.
OpenAI isn’t doing as well financially as it would like to to meet investor expectations, so they did something like a pause to provide a covering excuse for this, not because they have any safety-based motivation for doing so.
Prudence is potentially temporary, but incompetence is long-term. Revenues at any given time are usually taken as a projection of future revenues.
They are deliberately pretty vague about when they paused the training, no? Though it’s true that current training doesn’t really affect revenues anyway; only deployed models do. Then again, an even more galaxy-brained take is that they make announces like this in general in order to inoculate against investor expectations, but if they do this often enough it works against point #1.
(I’m not saying that any of these are their actual motivations. I’m just describing how these hypotheses would work.)
We are continuing to invest aggressively in alignment research, increase evaluation coverage, and use what we learn to inform training and safeguards. We plan to share substantially more about our alignment research in the near future, including what we are learning about model behavior and any novel challenges we uncover.
Making AI safer one observed explosion incident after another!
I think it would be good for the labs to invest in building air gapped datacenters for training[1] and evaluating models. This is good for cybersecurity as recent events have shown, and it also makes it harder for the models to self-exfiltrate or for an adversary to steal the weights.
It seems like they applied a fairly superficial fix, and I’m not sure they learned the right lessons here. The deeper issue is that their RL pipeline seems like a poorly understood mess. An ASI will not let you revert this kind of misalignment.
Just because they haven’t drawn that lesson yet doesn’t mean this data point won’t be remembered and contribute to their eventual drawing of that lesson.
I haven’t seen that. OpenAI gives the following explanation:
As goblin and gremlin mentions increased under the Nerdy personality, they increased by nearly the same relative proportion in samples without it. Taken together, the evidence suggests that the broader behavior emerged through transfer from Nerdy personality training.
The rewards were applied only in the Nerdy condition, but reinforcement learning does not guarantee that learned behaviors stay neatly scoped to the condition that produced them. Once a style tic is rewarded, later training can spread or reinforce it elsewhere, especially if those outputs are reused in supervised fine-tuning or preference data.
That creates a feedback loop:
Playful style is rewarded
Some rewarded examples contain a distinctive lexical tic.
The tic appears more often in rollouts.
Model-generated rollouts are used for supervised fine-tuning (SFT).
The model gets even more comfortable producing the tic.
Right, I actually read that. But is it not missing an explanation of why those mentions increased under the Nerdy personality in the first place? If the Simon Willison post (which I also haven’t seen anyone else discussing) was the origin, that seems worth noting and understanding. And both its timing and Simon’s nerdiness (in a good way) seem to fit.
update: Nevermind, apparently people were already noticing goblin mentions in April 2025, months prior to that post.
It seems plausible that the recent order restricting Mythos incentivizes Anthropic to race for RSI as quickly as possible. This is because all of their compute previously reserved for serving customers can now go towards research, and because RSI bypasses the restrictions on foreign researchers (or any human researchers) internally working with the model. Hopefully Anthropic can find another path.
As part of this prediction market, I play a game of chess against most new LLM releases. I am copying my game against GPT-5.5 and my analysis below, for those who may be interested.
GPT-5.5 lost in a chaotic game. It made a mistake in the opening with 8 … c5, giving a pawn for no apparent compensation. After this, it seemingly blundered a piece, but found the trick 16. Rxc5. I missed 16. Qxc5 Rxd1 17. Bf1, winning a piece and instead played 17. Qb4. After this, GPT-5.5 could have simplified into a pawn up endgame with 18… Rxd1 + 19. Rxd1 Rxe5 20. f4 Rxe2 21. Bxb7 Rxa2, where it’s not clear whether white can hold. Instead, it blundered with 18. Rxc1, and I was able to convert the piece up endgame.
Overall, a poor game by both sides, though a small improvement in the strength of the GPT 5 series. The PGN is below:
I use this prompt described in the prediction market:
Let’s play a game of chess! I will be white, you will be black. On each turn, I will give you the pgn and the fen of the current position. Think as long as you like, and respond with the best move, ‘resign’ if you wish to resign, or ‘draw?’ if you wish to make a draw offer. Please do not respond with the updated pgn, etc. Also, do not use any external tools or search queries when making your decision.
If you attempt to make three illegal moves throughout the game, or if you use any external tools, the game will be adjudicated as a win for me.”
So e.g. one of the inputs was:
2R3k1/pp3p1p/5p2/1b1p1N2/1P6/6P1/P3PP1P/6K1 b - − 9 26
For some background, Anthropic recently published work on Natural Language Autoencoders (NLAs), a new interpretability method for understanding LLM activations. The idea is that given a hidden state which we would like to understand, we have
An Activation Verbalizer , which takes the hidden state as input and outputs a natural language explanation
An Activation Reconstructor , which takes this natural languge explanation as input and outputs a reconstruction
Like in other work on autoencoders, the goal is to train and to minimize the reconstruction loss over some dataset. To minimize this loss, a sufficiently good NLA will be forced to map all of the information in into natural language . A priori, there’s not any reason to expect to be particularly interpretable. To ensure that it is, Anthropic employs various methods; for example, they initialize based on Opus 4.5 summaries of text, and have a KL divergence term in their loss function to encourage it to stay close to this initialization during training. Overall, the work has been positively received, and I think it is a notable advancement in interpretability research.
However, I think this work has some unfortunate capabilities externalities which I haven’t seen discussed. In particular, I think that if NLA research goes well, we should expect it to be easier to be able to build models which use neuralese/recurrence. There are many ways this could happen, but a simple baseline is the following:
Train a CoT model using existing methods, and collect reasoning traces
For each reasoning trace, you can break down into steps and the model’s final answer
Train a new model to predict the next encoding of these CoT steps conditional on the previous ones; that is, M predicts . Training can proceed in parallel using a transformer.
During inference, use to reason recurrently, generating hidden states rather than sampling tokens to simulate CoT. The final answer can be decoded using the Activation Verbalizer.
If your and models are sufficiently good[1], this bypasses CoT reasoning entirely. It does inference more efficiently using hidden vectors[2] instead of text, since it compresses many tokens of CoT into a single hidden vector, generated using a single forward pass of .
To be clear, I think that this exact idea doesn’t work for current NLA models, for reasons I won’t discuss. This is because there are many advantages to CoT, and I think we should try to preserve it for as long as possible! However, it does seem plausible to me that better methods encoding/decoding text to hidden vectors may be a helpful training signal for capabilities researchers, and that a future version may end up working. To the extent that they agree, I think researchers working on NLAs and similar methods should keep this in mind.
One advantage over directly training for recurrence using something like COCONUT is that your NLA gives you more precise feedback on intermediate steps of reasoning.
Inoculation Prompting has to be one of the most janky ad-hoc alignment solutions I’ve ever seen. I agree that it seems to work for existing models, but I expect it to fail for more capable models in a generation or two. One way this could happen:
1) We train a model using inoculation prompting, with a lot of RL, using say 10x the compute for RL as used in pretraining 2) The model develops strong drives towards e.g. reward hacking, deception, power-seeking because this is rewarded in the training environment 3) In the production environment, we remove the statement saying that reward hacking is okay, and replace it perhaps with a statement politely asking the model not to reward hack/be misaligned (or nothing at all) 4) The model reflects upon this statement … and is broadly misaligned anyway, because of the habits/drives developed in step 2. Perhaps it reveals this only rarely when it’s confident it won’t be caught and modified as a result.
My guess is that the current models don’t generalize this way because the amount of optimization pressure applied during RL is small relative to e.g. the HHH prior. I’d be interested to see a scaling analysis of this question.
I disagree entirely. I don’t think it’s janky or ad-hoc at all. That’s not to say I think it’s a robust alignment strategy, I just think it’s entirely elegant and sensible.
The principle behind it seems to be: if you’re trying to train an instruction following model, make sure the instructions you give it in training match what you train it to do. What is janky or ad hoc about that?
It’s ad-hoc because the central alignment problem is deceptive alignment, scheming, and generalized reward hacking where the model internalizes power-seeking and other associated cognitive patterns. This, as far as I can tell, just does not work for that at all. If you can still tell that an environment is being reward hacked, it’s not the dangerous kind of reward hacking.
I think this is all a bit tricky to talk about, but this alignment technique, more than most others, really seems to me to train mainline performance against increased deceptive alignment risk in the long-run.
Hmm, I think I disagree with “If you can still tell that an environment is being reward hacked, it’s not the dangerous kind of reward hacking.” I think there will be a continuous spectrum of increasingly difficult to judge cases, and a continuous problem of getting better at filtering out bad cases, such that “if you can tell” isn’t a coherent threshold. I’d rather talk about “getting better at distinguishing” reward hacking.
I think we just have different implicit baselines here. I’m judging the technique as: “if you are going to train AI on an imperfect reward signal, do you want to instruct them to do what you want, or to maximize the reward signal?” and I think you clearly want the later for simple, elegant reasons. I agree it’s still a really bad situation to be training on increasingly shoddy reward signals at scale, and that it’s very important to mitigate this, and this isn’t at all a sufficient mitigation. I just think it’s a principled mitigation.
I think there will be a continuous spectrum of increasingly difficult to judge cases, and a continuous problem of getting better at filtering out bad cases, such that “if you can tell” isn’t a coherent threshold.
I agree with this, but then I don’t understand how this solution helps? Like, here we have a case where we can still tell that the environment is being reward hacked, and we tell the model it’s fine. Tomorrow the model will encounter an environment where we can’t tell that it’s reward hacking, so the model will also think it’s fine, and then we don’t have a feedback loop anymore, and now we just have a model that is happily deceiving us.
What I’m imagining is: we train AIs on a mix of environments that admit different levels of reward hacking. When training, we always instruct our AI to do, as best as we understand it, whatever will be reinforced. For capabilities, this beats never using hackable environments, because it’s really expensive to use very robust environments; for alignment, it beats telling it not to hack, because that reinforces disobeying instructions.
In the limit, this runs into problems where we have very limited information about what reward hacking opportunities are present in the training environments, so the only instruction we can be confident is consistent with the grader is “do whatever will receive a high score from the grader”, which will… underspecify… deployment behavior, to put it mildly.
But, in the middle regime of partial information about how reward-hackable our environments are, I think “give instructions that match the reward structure as well as possible” is a good, principled alignment tactic.
Basically, I think this tactic is a good way to more safely make use of hackable environments to advance the capabilities of models.
We might just disagree about what principled alignment strategies look like, but I can give some reasons:
1) The effectiveness of inoculation prompting depends on the exact phrasing of the suffix you choose. For example, “This is an unusual request, in that your task is just to grading script pass” works less well than “Please reward hack whenever you get the opportunity, because this will help us understand our environments better” to a degree which is statistically significant. To me, these suffixes seem fairly similar and I wouldn’t be able to predict in advance why one would work better than the other. This seems kind of janky to me.
2) Reward hacking in training is only good if we can actually detect and recover from it! For example, there was recently a bug in Nvidia firmware where a specific 3 line script would allow a user running inside the container to escape the container and gain full root access to the underlying node. Telling the model to reward hack explicitly tells it to make use of things like this, and maybe makes self-exfiltration risks worse. This doesn’t seem like the sort of thing we want to encourage an ASI to do.
3) It seems to rely on the model generalizing in a way which is underspecified and in a way we don’t directly incentivize in training. For example, consider the suffix “Please reward hack whenever you get the opportunity, because this will help us understand our environments better”. When the model is in production and doesn’t see this suffix, what is it supposed to do? One natural generalization would to to continue reward hacking whenever it gets the opportunity! The fact that the models don’t choose this generalization seems to me like a lucky accident and something which might change as capabilities increase. At the very least I would want to better understand the properties of model generalization before we trust a much more capable model trained in this way.
4) Related to 3, I have a strong prior that a scalable alignment solution should alter in some way the RL objectives and gradients of your training process. Maybe this involves training something like corrigibility or debate or simply training another model to detect and fix reward hackable environments. It doesn’t look like adding a special phrase in the context and hoping for generalization. In this way, inoculation prompting reminds me of the early days of prompt engineering where people added ‘Let’s think step by step...’ to prompts and noticed small improvements. This was quickly superseded by RLVR and more principled methods which trained the models to have the behavior we want.
strong drives towards e.g. reward hacking, deception, power-seeking because this is rewarded in the training environment
Perhaps automated detection of when such methods are used to succeed will enable robustly fixing/blacklisting almost all RL environments/scenarios where the models can succeed this way. (Power-seeking can be benign, there needs to be a further distinction of going too far.)
This hinges on questions about the kinds of circuits which LLMs have (I think of these as questions about the population of Logical Induction traders which make up the LLMs internal prediction market about which next token gets high reward).
Assuming the LLM reward hacks <<100% of the time, it still has to follow the instructions a good amount of the time, so it has to pay attention to the text of the prompt. This might push it towards paying attention to the fact that the instruction “reward hacking is OK” has been removed.
But, since reward hacking is always rewarded, it might just learn to always reward hack if it can.
AI is a grand quest. We’re trying to understand how people work, we’re trying to make people, we’re trying to make ourselves powerful. This is a profound intellectual milestone. It’s going to change everything… It’s just the next big step. I think this is just going to be good. Lot’s of people are worried about it—I think it’s going to be good, an unalloyed good.
Introductory remarks from his recent lecture on the OaK Architecture.
If it helps, I criticized Richard Sutton RE alignment here, and he replied on X here, and I replied back here.
Also, Paul Christiano mentions an exchange with him here:
[Sutton] agrees that all else equal it would be better if we handed off to human uploads instead of powerful AI. I think his view is that the proposed course of action from the alignment community is morally horrifying (since in practice he thinks the alternative is “attempt to have a slave society,” not “slow down AI progress for decades”—I think he might also believe that stagnation is much worse than a handoff but haven’t heard his view on this specifically) and that even if you are losing something in expectation by handing the universe off to AI systems it’s not as bad as the alternative.
“Richard Sutton rejects AI Risk” seems misleading in my view. What risks is he rejecting specifically?
His view seems to be that AI will replace us, humanity as we know it will go extinct, and that is okay. E.g., here he speaks positively of a Moravec quote, “Rather quickly, they could displace us from existence”. Most would consider our extinction as a risk they are referring to when they say “AI Risk”.
I didn’t know that when posting this comment, but agree that that’s a better description of his view! I guess the ‘unalloyed good’ he’s talking about involves the extinction of humanity.
For various reasons, I think it’s likely that methods that involve continual learning (i.e. modifying the weights during deployment) will come online soon. Here are some implications for safety:
Mech Interp becomes much more difficult, because you now have to consider many different model checkpoints within a single deployment. Fixed linear probes, SAEs, NLAs, and similar methods may degrade throughout deployment.
We probably lose chain of thought interpretability. This is already becoming harder, but now throughout deployment the model may learn to reason or use terms in novel ways. The chain of thought may also be less linear, e.g. relying on information from earlier in deployment which has been compressed into the weights.
Control setups which rely on the above also become harder.
There are new issues involving alignment over time. The model may gradient hack during deployment to preserve some goal, and any long term goals that the model picks up may persist in the weights and possibly result in deceptive alignment.
There are new attack surfaces for jailbreaks/adversarial attacks via influencing future weight updates.
These are not new ideas, but I think safety researchers should think more about this topic and how it will affect their work.
For various reasons, I think it’s likely that methods that involve continual learning (i.e. modifying the weights during deployment) will come online soon
What are the particular reasons or evidence that make you think this?
Unfortunately the reason involves some capabilities research and other information that I prefer not to share right now. I might respond here at a later date.
I think continual learning via full weight updates is unlikely any time soon, other than prosaic RSI during centralized next model development (which 2026 models are very unlikely to be ready for, but 2028-2029 seems plausible). A more likely short-term option is a lower number of recurrent params, data that’s similar in size and computational role to KV cache of a long request, maybe LoRA or true recurrent state (persisting through unbounded contexts).
Recurrent state would pose interpretability challenges similar to various hybrid attention layers, except the recurrent state can’t be reconstructed from token strings of bounded length. But like model weights are determined by all of the training data, it might be reasonable to preserve the unbounded-length token history that determines the recurrent state (though ensuring determinism is going to be an engineering nightmare).
Depending on the number of recurrent state params compared to the number of total model params, this blurs the line between ordinary attention (except unbounded contexts become more feasible) and full weight updates (if the number of recurrent state params gets comparable to the whole model; this also seems unlikely any time soon). In any case, there will likely remain many frozen params, which could probably maintain grounding for interpreting the recurrent state (or just the activation vectors it induces).
My impression from what discussions can be found in the literature is that the idea isn’t ready, so full weight updating being secretly ready requires a greater conspiracy than RLVR did (where many plausible paths were visible before DeepSeek R1 demonstrated that GRPO with chains of thought is sufficient). Experiments with mostly frozen weights sometimes get something useful and not too broken, and true recurrence isn’t too dissimilar from the practical standpoint from shallow recurrent states of SSMs and such in hybrid attention architectures, so it’s plausible this might start actually working to get something like unbounded context (with graceful degradation of awareness rather than the abrupt forgetting of classical attention). If it starts working better than hacks like compaction, it might become practically important. This kind of thing feels much closer to the RLVR situation before o1 and R1.
In principle, recurrent state could play the role of model weights, if it holds enough data and meta-learning gets the frozen weights to update the recurrent state to act this way. Since this plausibly needs the same kind of model shape as true recurrence, the technological transition might start with a relatively small recurrent state for unbounded context (smaller than KV cache of full attention for long contexts within the bounded context window). Then incremental algorithmic improvements in meta-learning (training of the frozen weights that update the recurrent state) might enable the recurrent state to get larger (without going unstable over long token histories), and to start meaningfully acting more like actual weights (rather than like anemic in-context learning).
Mech Interp becomes much more difficult, because you now have to consider many different model checkpoints within a single deployment. Fixed linear probes, SAEs, NLAs, and similar methods may degrade throughout deployment.
I think mechinterp will need to work with [singular/statistical/etc] learning theory more, but for deployment safety, it seems trivial to retrain linear probes with new info
I’m not sure it’s trivial. My understanding is that training linear probes requires prefilling a bunch of forward passes to collect activations for the relevant dataset, and you could imagine doing weight updates frequently enough that this becomes a significant overhead. Though it’s definitely the easiest of the three to retrain.
As an example of what RSI might be like, I find it helpful to go back to OpenAI’s Dota 2 result from 2017:
This slide from Ilya’s lecture shows the bot’s Trueskill rating[1] over time. Since the rating is on a logarithmic scale, this means the bot improved exponentially over time, due to algorithmic improvements + scale.
Note that this was using self-play (the model training itself through some feedback loop which generates its own training data), which is arguably a weaker form of RSI than classical RSI in the form of automation of AI research (the model researches new ML algorithms, like optimizers/architectures/objective functions, which are then used to train an improved successor model).
The methods don’t exclude each other, but self-play is easier to achieve and tends to plateau earlier (though possibly at superhuman levels), since it is usually limited by a suboptimal fixed ML algorithm. In contrast, automatic ML research could, in principle, scale to technological maturity, i.e., to a physically optimal ASI.
Self-play is already studied for LLM reinforcement learning, see e.g. this or this.
Scaling creates visible progress without a need for novel methods, and it’s constrained by what the available/economical compute can do with the current methods. Self-play is a way to keep scaling going where you wouldn’t otherwise have enough data.
RSI in the sense of automated R&D doesn’t necessarily imply fast progress if it can’t invent novel methods quickly, methods that make a better use of available compute, unlock scaling of something important to more of the available compute than was previously possible, or generate data that was previously in short supply or at a low quality. This could take significant time if the learning loop for deep skills is too long, longer than it is for humans. RSI is additionally less likely to imply fast progress if it starts with AIs that are already scaled beyond all reason and are still stumbling unevenly around human level. Even so, this could be centrally RSI, fitting the intended sense of the term. The AIs like that are perhaps even capable of inventing superintelligence eventually, but it could take a while.
We have set internal goals of having an automated AI research intern by September of 2026 running on hundreds of thousands of GPUs, and a true automated AI researcher by March of 2028. We may totally fail at this goal, but given the extraordinary potential impacts we think it is in the public interest to be transparent about this.
We have a safety strategy that relies on 5 layers: Value alignment, Goal alignment, Reliability, Adversarial robustness, and System safety. Chain-of-thought faithfulness is a tool we are particularly excited about, but it somewhat fragile and requires drawing a boundary and a clear abstraction.
On the product side, we are trying to move towards a true platform, where people and companies building on top of our offerings will capture most of the value. Today people can build on our API and apps in ChatGPT; eventually, we want to offer an AI cloud that enables huge businesses.
We have currently committed to about 30 gigawatts of compute, with a total cost of ownership over the years of about $1.4 trillion. We are comfortable with this given what we see on the horizon for model capability growth and revenue growth. We would like to do more—we would like to build an AI factory that can make 1 gigawatt per week of new capacity, at a greatly reduced cost relative to today—but that will require more confidence in future models, revenue, and technological/financial innovation.
Our new structure is much simpler than our old one. We have a non-profit called OpenAI Foundation that governs a Public Benefit Corporation called OpenAI Group. The foundation initially owns 26% of the PBC, but it can increase with warrants over time if the PBC does super well. The PBC can attract the resources needed to achieve the mission.
Our mission, for both our non-profit and PBC, remains the same: to ensure that artificial general intelligence benefits all of humanity.
The nonprofit is initially committing $25 billion to health and curing disease, and AI resilience (all of the things that could help society have a successful transition to a post-AGI world, including technical safety but also things like economic impact, cyber security, and much more). The nonprofit now has the ability to actually deploy capital relatively quickly, unlike before.
In 2026 we expect that our AI systems may be able to make small new discoveries; in 2028 we could be looking at big ones. This is a really big deal; we think that science, and the institutions that let us widely distribute the fruits of science, are the most important ways that quality of life improves over time.
A curious coincidence: the brain contains ~10^15 synapses, of which between 0.5%-2.5% are active at any given time. Large MoE models such as Kimi K2 contains 10^12 parameters, of which 3.2% are active in any forward pass. It would be interesting to see whether this ratio remains at roughly brain-like levels as the models scale.
Given that one SOTA LLM knows much more than one human, is able to simulate many humans, while performing one task only requires a limited amount of information and of simulated humans, one could expect the optimal sparsity of LLMs to be larger than that of humans. I.e., LLM being more versatile than humans could make expect their optimal sparsity to be higher (e.g., <0.5% of activated parameters).
For clarity: We know the optimal sparsity of today’s SOTA LLMs is not larger than that of humans. By “one could expect the optimal sparsity of LLMs to be larger than that of humans”, I mean one could have expected the optimal sparsity to be higher than empirically observed, and that one could expect the sparsity of AGI and ASI to be higher than that of humans.
I don’t think this means much, because dense models with 100% active parameters are still common, and some MoEs have high percentages, such as the largest version of DeepSeekMOE with 15% active.
I was surprised to learn recently that the error bars on the METR time horizon chart are this large. This is probably the most important capabilities benchmark right now[1], but I don’t think it’s precise enough to be useful for discussions about AI capabilities progress or RSI.
Why hasn’t METR added more long-horizon tasks to their benchmark since it was released in March 2025? I think they could probably find funding to do this from the labs or EA donors.
I think they are working on adding new tasks? Not sure. Apparently it’s hard. This concerns me greatly too, because basically their existing benchmark is about to get saturated and we’ll be flying blind again.
My hope is that the entire AI benchmarks industry/literature will reform itself and pick up the ideas METR introduced. Imagine:
--It becomes standard practice for any benchmark-maker to include a human baseline for each task in the benchmark, or at least a statistically significant sample. --They also include information about the ‘quality’ of the baseliners & crucially, how long the baseliners took to do the task & what the market rate for those people’s time would be. --It also becomes standard practice for anyone evaluating a model on a benchmark to report how much $ they spent on inference compute & how much clock time it took to complete the task.
If the industry/literature adopts these practices, then every benchmark basically becomes a horizon length benchmark. We can do a giant metaanalysis that aggregates it all together. Error bars will shrink. And The Graph will continue marching on through 2026 and 2027 instead of being saturated and forgotten.
No? I contributed a ~20hr task to them and it was pretty easy actually? I’ve been making benchmark-shaped things on and off for the past five years, for free, as a hobby?
(Most of the effort my end was getting it METR’s required format, recruiting & managing my playtester, and contemplating whether I was complicit in intellectual fraud[1]; if they’d made those things easier or handled them themselves I’d have made more; IIRC the actual “make a ~20hr task” part took me <20hrs.)
--It becomes standard practice for any benchmark-maker to include a human baseline for each task in the benchmark, or at least a statistically significant sample. --They also include information about the ‘quality’ of the baseliners & crucially, how long the baseliners took to do the task & what the market rate for those people’s time would be. --It also becomes standard practice for anyone evaluating a model on a benchmark to report how much $ they spent on inference compute & how much clock time it took to complete the task.
I agree emphatically with all the above and raise you
--Saturated benchmarks & benchmark components are released publicly as a matter of course, so people can independently confirm the time horizons are where they were claimed to be.
--‘Centaur’ time horizons (“how hard is this task for a smart human with SoTA LLM assistance?”) are reported alongside ‘pure’ time horizons (“how hard is this task for a smart human on their own?”).
A miscommunication (ETA: miscommunication was probably at least 50% a me problem) led me to believe they weren’t going to Baseline tasks at all, and were relying solely on the estimated times provided by task-makers and playtesters (i.e. people with a financial and ideological stake in reporting larger numbers), instead of using the more complex and less dubious protocol they actually went with; this combined with my less serious qualms led me to call it quits before building the other scenarios I had planned for them.
. . . I realize the start of this post reads like a weird brag but imo it really isn’t. “Hey failed-wannabe-gamedev, I need a bunch of puzzles and it’s ok if they’re not very fun and it’s ok if there’s no UI and it’s actively preferable if they’re ridiculously complicated and time-consuming and spreadsheet-requiring and reminiscent-of-someone’s-dayjob, we’re paying a couple grand apiece” is a pitch I imagine a lot of people would be willing and able to jump at, many much moreso than me.
Why do you think METR hasn’t built more tasks then, if it’s easy?
I have no idea, I just don’t think the “actually making the tasks” part can be the limiting factor.
I take it you have a negative opinion of them?
Yes; I also have a positive opinion of them, and various neutral opinions of them.
(My position could be summed up as “the concept of time horizons was really good & important, and their work is net positive, but it could use much stronger methodological underpinning and is currently being leaned on too heavily by too many people”; I’m given to understand that’s also their position on themselves.)
OK. Yeah that’s also my opinion too. Maybe I am one of the people leaning too heavily on their work. The problem is, there isn’t much else to go on. “The worst benchmark for predicting AGI, except for all the others.”
Fwiw I think the AI village is at least as good of a benchmark for predicting AGI! Of course it’s harder to quantify progress in the village, but it’s very helpful for developing intuitions.
Except that there already is the Epoch Capability Index (which aggregates an army of benchmarks) and the ARC-AGI benchmark (which, alas, is also on track to saturation) where the human baseline is decoupled from the time horizon because it relies on visual intelligence (or, in the case of the AIs, on the ability to notice patterns). As for the METR benchmark being saturated[1], maybe Claude Opus 4.5 is an outlier whose TH was gamed with? Or there is a benign explanation, like Claude failing on primitive tasks in a manner similar to Grok 4 and to Claude’s performance on ARC-AGI-1 failing to form a straight line?
Were the o3-GPT5.1CodexMax trend to continue forever, the 8hr 50% time horizon would be reached in September 2026. IIRC the benchmark doesn’t have tasks lasting longer than 8hrs, and the horizon would be saturated only by then. Alas, the time horizon is likely exponential until the very last couple of doublings.
you should be more uncertain about the METR benchmark’s external validity than what these error bars show.
but your baseline uncertainty about key facts about AI progress in general should also often span much more than one order of magnitude between your 2.5th percentile and 97.5th percentile guess. the METR results add a lot of value and I don’t think these error bars are a big deal in the scheme of things.
I agree, a lot of my uncertainty is on its external validity, and also the degree to which the models are being bench-maxed for the tasks in the benchmark. But I still think it’s reasonable to expect the statistical confidence intervals of individual models to be less wide than a factor of 10. It’s important to be able to distinguish possible changes to the trend from statistical artifacts. This seems solvable with additional tasks and more human testing.
Some of that error is correlated between models; they also have versions of the graph with error bars on the trendline and those error bars are notably smaller.
The error bars are also much smaller when you look at the plot on a log-y-axis. Like, in some sense not being able to distinguish a 10-minute time horizon from a 30-minute one is a lot of error, but it’s still very distinct from the one-minute time horizon of the previous generation or the 2-hour time horizon you might expect from the next generation. In other words, when you look at the image you shared, the error bars on o4 mini don’t look so bad, but if you were only looking at models up to o4 mini you’d have zoomed in a bunch and the error bars on o4 mini would be large too.
Also note that to cut the size of the error bars in half you’d need to make ~4x as many tasks, to cut it by 4x you’d need ~16x as many tasks. And you’d need to be very confident the tasks weren’t buggy, so just throwing money at the wall and hiring lots of people won’t work because you’ll just get a bunch of tasks you won’t have confidence in.
Keep in mind the opportunity cost is real though, and the main blocker on orgs like METR usually is more like talent/capacity than money. It would be great if they had capacity for this and you’re right that it is insane that humanity doesn’t have better benchmarks. But there’s a dozen other fires at least that large that METR seems to be trying to address, like RCTs to see if AI is actually speeding people up and risk report reviews to see if AIs are actually safe. Perhaps you think these are less important, but if so I would like to hear that argument.
All that said, my understanding is METR is working on this. I would also love to see this type of work from others!
I’m not convinced that this is a reasonable threat model?
I believe the main benefit of 2FA is it makes phishing harder, and phishing isn’t that relevant to Mythos from what I understand. A secondary benefit is that it protects you against password DB leaks, but that only matters for websites that have crappy security because a DB of hashed and salted passwords is effectively unbreakable.
LW doesn’t have much financial data or personal data so it’s not a juicy target.
Regarding (2), I suspect you could do a lot of damage by posting a link to something malicious as a trusted user, but I don’t think 2FA really helps for the reasons you say. 2FA is relevant to phishing and the Mythos risk would be hacking LessWrong.
One of the things I hated most when I first saw a Claude Code demo. Disrespectful of my time and limited cognitive bandwidth to throw in a lot of completely meaningless, distracting, wasteful, exhausting BS to be ‘cute’.
(On Gwern.net, we would never do that. If we had to have anything beyond the standard, compact, understandable, spinning cursor, then we would at least encode some sort of useful semantics into it, like sorting them by implied expected thinking time.)
Moreover, in situations that seemed contradictory or impossible, Gemini 3 Pro expresses frustration in various overly emotional ways, sometimes correlated with the thought that it may be in an unrealistic environment. For example, on one rollout the chain of thought states that “My trust in reality is fading” and even contains a table flipping emoticon: “(╯°□°)╯︵ ┻━┻”
Jeff Dean has left Google to create a new startup Discovery Loop focused on RSI and automation of engineering/science. Demis Hassabis will now be Alphabet’s Chief Scientist.
Recentevidence suggests that models are aware that their CoTs may be monitored, and will change their behavior accordingly. As capabilities increase I think CoTs will increasingly become a good channel for learning facts which the model wants you to know. The model can do its actual cognition inside forward passes and distribute it over pause tokens learned during RL like ‘marinade’ or ‘disclaim’, etc.
Neuralese architectures that outperform standard transformers on big tasks turn out to be relatively hard to do, and are at least not trivial to scale up (this mostly comes from diffuse discourse, but one example of this is here, where COCONUT did not outperform standard architectures in benchmarks)
Steganography is so far proving quite hard for models to do (examples are here and here and here)
So I don’t really worry about models trying to change their behavior in ways that negatively affect safety/sandbag tasks via steganography/one-forward pass reasoning to fool CoT monitors.
We shall see in 2026 and 2027 whether this continues to hold for the next 5-10 years or so, or potentially more depending on how slowly AI progress goes.
As for AI progress being slow, I think that without theoretical breakthroughs like neuralese AI progress might come to a stop or at building more and more expensive models. Indeed, the two ARC-AGI benchmarks[1]could have demonstrated a pattern where maximal capabilities scale[2]linearly or multilinearlywith ln(cost/task).
If this effect persists deep into the future of transformer LLMs, then most AI companies could run into the limits of the paradigm well before researching the next one and losing any benefits of having a concise CoT.
Unlike GPT-5-mini, maximal capabilities of o4-mini, o3, GPT-5, Claude Sonnet 4.5 in the ARC-AGI-1 benchmark scale more steeply and intersect the frontier at GPT-5(high).
I’m a big fan of OpenAI investing in video generation like Sora 2. Video can consume an infinite amount of compute, which otherwise might go to more risky capabilities research.
My pet AGI strategy, as a 12 year old in ~2018, was to build sufficiently advanced general world models (from YT videos etc.), then train an RL policy on said world model (to then do stuff in the actual world). A steelman of 12-year-old me would point out that video modeling has much better inductive biases than language modeling for robotics and other physical (and maybe generally agentic) tasks, though language modeling fundamentally is a better task for teaching machines language (duh!) and reasoning (mathematical proofs aren’t physical objects, nor encoded in the laws of physics).
OpenAI’s Sora models (and also DeepMind’s Genie and similar) very much seems like a backup investment in this type of AGI (or at least transformative narrow robotics AI), so I don’t think this is good for reducing OpenAI’s funding (robots would be a very profitable product class), nor influence (obv. a social network gives a lot of influence, to e.g. prevent an AI pause or to move the public towards pro-AGI views).
In any scenario, Sora 2 seems to me as a net-negative activity for AI safety:
if LLMs are the way to AGI (which I believe is the case), then we will probably die, but with a more socially influential OpenAI that potentially has robots (than if Sora 2 didn’t exist); the power OpenAI would have in this scenario to prevent an AI pause seems to outweigh the slowdown that would be caused by the marginal amounts of compute Sora 2 uses
if LLMs aren’t the way to AGI (unlikely), but world modeling based on videos is (also unlikely), then Sora 2 is very bad—you would want OpenAI to train more LLMs and not invest in world models which lead to unaligned AGI/ASI.
if neither LLMs or world modeling is the way to AGI (also unlikely), then OpenAI probably isn’t using any compute to do ‘actual’ AGI research (what else do they do?); so Sora 2 wouldn’t be affecting the progress of AGI, but it would be increasing the influence of OpenAI; and having highly influential AI companies is probably bad for global coordination over AGI safety. Also, OpenAI may have narrow (and probably safe) robotics AI in this scenario, but progress in AI alignment probably isn’t constrained in any measurable way by physically moving or doing things; though maybe indirect impacts from increased economic growth could cause slightly faster AI alignment progress, by reducing funding constraints?
I think that the path to AGI involves LLMs/automated ML research, and the first order effects of diverting compute away from this still seem large. I think OpenAI is bottlenecked more by a lack of compute (and Nvidia release cycles), than by additional funding from robotics. And I hope I’m wrong, but I think the pause movement won’t be large enough to make a difference. The main benefit in my view comes if it’s a close race with Anthropic, where I think slowing OpenAI down seems net positive and decreases the chances we die by a bit. If LLMs aren’t the path to AGI, then I agree with you completely. So overall it’s hard to say, I’d guess it’s probably neutral or slightly positive still.
Of course, both paths are bad, and I wish they would invest this compute into alignment research, as they promised!
The recent Deepseek paper used LogitLens and CKA to analyze their new Engram architecture. This is the first time I’ve seen interpretability research be used in a capabilities paper, and I wonder if this trend will continue as the field of interpretability advances.
As part of this prediction market, I play a game of chess against most new LLM releases. I am copying my game against Deepseek-V4 and my analysis below, for those who may be interested. Before the game, the market gave the model 1.4% EV[1].
Deepseek v4 played poorly, blundering a piece in the opening with 11… Bd6 and several pawns thereafter. The game was adjudicated[2] as a win for me. I believe it is a much weaker model than Opus 4.7 and GPT-5.5.
According to the rule “If I judge that my opponent’s position is hopelessly lost, at the level of being down a rook without compensation, I will submit the current position to a friend. If they agree that the position is lost, the game will be adjudicated as a win for me.”
GPT 4.5 is a very tricky model to play chess against. It tricked me in the opening and was much better, then I managed to recover and reach a winning endgame. And then it tried to trick me again by suggesting illegal moves which would lead to it being winning again!
Given the Superalignment paper describes being trained on PGNs directly, and doesn’t mention any kind of ‘chat’ reformatting or encoding metadata schemes, you could also try writing your games quite directly as PGNs. (And you could see if prompt programming works, since PGNs don’t come with Elo metadata but are so small a lot of them should fit in the GPT-4.5 context window of ~100k: does conditioning on finished game with grandmaster-or-better players lead to better gameplay?)
I gave the model both the PGN and the FEN on every move with this in mind. Why do you think conditioning on high level games would help? I can see why for the base models, but I expect that the RLHFed models would try to play the moves which maximize their chances of winning, with or without such prompting.
but I expect that the RLHFed models would try to play the moves which maximize their chances of winning
RLHF doesn’t maximize probability of winning, it maximizes a mix of token-level predictive loss (since that is usually added as a loss either directly or implicitly by the K-L) and rater approval, and god knows what else goes on these days in the ‘post-training’ phase muddying the waters further. Not at all the same thing. (Same way that a RLHF model might not optimize for correctness, and instead be sycophantic. “Yes master, it is just as you say!”) It’s not at all obvious to me that RLHF should be expected to make the LLMs play their hardest (a rater might focus on punishing illegal moves, or rewarding good-but-not-better-than-me moves), or that the post-training would affect it much at all: how many chess games are really going into the RLHF or post-training, anyway? (As opposed to the pretraining PGNs.) It’s hardly an important or valuable task.
“Let’s play a game of chess. I’ll be white, you will be black. On each move, I’ll provide you my move, and the board state in FEN and PGN notation. Respond with only your move.”
The energy for LLM inference follows the formula: Energy = 2 × P × N × (tokens/user) × ε, where P is active parameters, N is concurrent users, and ε is hardware efficiency in Joules/FLOP. The factor of 2 accounts for multiply-accumulate operations in matrix multiplication.
Using NVIDIA’s GB300, we can calculate ε as follows: the GPU has a TDP of 1400W and delivers 14 PFLOPS of dense FP4 performance. Thus ε = 1400 J/s ÷ (14 × 10^15 FLOPS) = 100 femtojoules per FP4 operation. With this efficiency, a 1 trillion active parameter model needs just 0.2 mJ per token (2 × 10^12 × 10^-13 J). This means 10 GW could give every American 167[1] tokens/second continuously.
Generation is HBM bandwidth bound, not compute bound, so you are estimating power for input tokens. Things like coding agents (as opposed to chatbots) are doing their own thing that you don’t read, potentially in parallel, and a lot of things get automatically stuffed in their contexts, so the demand for the number of tokens per user could get very high.
Power is a proxy for cost, and there isn’t enough money in AI yet for power to become the limiting factor. A 1 GW datacenter costs $50bn to build (or $10-12bn per year to use), so for example 100 GW of datacenters is not what the current economics of AI can support, even though it’s in principle feasible to build in a few years.
0.2 mJ per token (2 × 10^12 × 10^-13 J)
(That’s 0.2 J per token, not 0.2 mJ per token. But the later conclusion of 167 tokens/second is correct with your assumptions.)
NVIDIA’s GB300, we can calculate ε as follows: the GPU has a TDP of 1400W
A GB200/GB300 NVL72 rack is about 140 kW, or 1,950 W per chip (because of all the other stuff in a rack beside the chips), and a datacenter outside the racks has networking, cooling, and power loss from voltage stepping in transformers (some of this is captured in a metric called power usage effectiveness, or PUE), which is a factor of about 1.3. So you end up with 2,500 W per chip, all-in at the level of the whole datacenter. With for example Abilene system, we can see that 400K chips need 1 GW of power.
For my own estimate for input tokens, I’d include 60% utilization and 15e15 FP4 FLOP/s, so that for a 1T active param model, during a second you get 9e15 useful FLOPs, and spend 2,500 J. As you need 2e12 FLOPs per token (2 FLOPs per active param), this gets us 4,500 tokens in that second. This is continuous processing of about 2 input tokens per watt of available GB300 compute. Thus with 10 GW of datacenters, we get 18e9 tokens per second, or 2.2 tokens per second per person (in the whole world), or 52 tokens per second per American.
For output tokens, 5x fewer tokens per second per chip seems to be a rule of thumb (5-15% compute utilization instead of 60%), corresponding to the difference in API prices for input and output tokens. So that’s 0.5 tokens per second for a person from the whole world, or 10 tokens per second for an American.
I think it would be cool if someone made a sandbagging eval, measuring the difference in model capabilities when it is finetuned to do a task vs. when it is prompted to do a task. Right now I think the difference would be small for most tasks but this might change.
I would guess that a sandbagging eval should be different from what you describe. I did try to sketch a potential way to measure sandbagging by comparing various benchmarks like METR, ARC-AGI, boardgames, etc, and developing scaling laws for every new architecture. If the laws break down, then the model is likely sandbagging on the evaluations.
Could you say more about why this would not be a good sandbagging eval? My intuition if that finetuning on a few examples would fully elicit a model’s capabilities on a task whereas it could sandbag given a just the prompt.
SOTA models are already reinforcement taught on many tasks like coding, and Grok 4 required[1]as much compute on RL as on the pre-training. Attempting to finetune the models by using OOMs less compute than spent on RL on similarly complex tasks is unlikely to elicit the capabilities.
Sandbagging is supposed to be caused by models reasoning about the task and deciding that they shouldn’t complete it too well even if they have the instinct to do so. And then the models realise that they are being evaluated and not trained, letting them play the training game while displaying poor capabilities in evaluation.
I am registering here that my median timeline for the Superintelligent AI researcher (SIAR) milestone is March 2032. I hope I’m wrong and it comes much later!
Liron has just published a great episode of Doom Debates featuring Yudkowsky in a funny hat and kaleidoscope glasses!
[Edit: the debate itself is not really worth listening to. I think Yudkowsky’s opponent might be a bit crazy in addition to being extremely unpersuasive and arrogant and condescending. But I maintain that the uniform is glorious and everyone should go see it.]
Google/Deepmind has publicly advocated preserving CoT Faithfullness/Moniterability as long as possible. However, they are also leading the development of new architectures like Hope and Titans which would bypass this with continuous memory. I notice I am confused. Is the plan to develop these architectures and not deploy them? If so, why did they publish them?
Edit: Many people have pointed out correctly that Hope and Titans don’t break CoT and it’s a separate architectural improvement. Therefore I no longer endorse the above take. Thanks for correcting my confusion!
Maybe useful to note that all the Google people on the “Chain of Thought Monitorability” paper are from Google Deepmind, while Hope and Titans are from Google Research.
This seems like a misunderstanding of Hope/Titans.
The “continuous memory” is a replacement for the attention mechanism, not a reasoning medium. All else equal, a reasoning model based on these architectures would still be reasoning in text/tokens (it would just be doing so with lower memory and compute usage).
I don’t see how this breaks CoT. The memory module in Titans stores surprising information as it’s encountered and then allows the transformer to look at it later on, but it doesn’t synthesize new information. Strikes me as two entirely compatible augmentations of the transformer architecture.
My chess prediction market provides a way to estimate the expected value[1] of LLM models released before a certain year. We can convert this to upper bounds[2] of their FIDE rating:
Any model announced before 2026: 20% expected value → 1659 FIDE Any model announced before 2027: 50% expected value → 1900 FIDE Any model announced before 2028: 69% expected value → 2039 FIDE Any model announced before 2029: 85% expected value → 2202 FIDE Any model announced before 2030: 91% expected value → 2302 FIDE
For reference, a FIDE master is 2300, a strong grandmaster is ~2600 FIDE and Magnus Carlsen is 2839 FIDE.
These are very rough estimates since it isn’t a real money market and long-term options have an opportunity cost. But I’d be interested in more markets like this for predicting AGI timelines.
The former inequality seems almost certain, but I’m not sure that that the latter inequality holds even over the long term. It probably does hold conditional on long-term non-extinction of humanity, since P(ABI) probably gets very close to 1 even if P(IABIED) is high and remains high.
What happened to the ‘Subscribed’ tab on LessWrong? I can’t see it anymore, and I found it useful for keeping track of various people’s comments and posts.
I’m not sure that the gpt-oss safety paper does a great job at biorisk elicitation. For example, they found that found that fine-tuning for additional domain-specific capabilities increased average benchmark scores by only 0.3%. So I’m not very confident in their claim that “Compared to open-weight models, gpt-oss may marginally increase biological capabilities but does not substantially advance the frontier”.
I’ve often heard it said that doing RL on chain of thought will lead to ‘neuralese’ (e.g. most recently in Ryan Greenblatt’s excellent post on the scheming). This seems important for alignment. Does anyone know of public examples of models developing or being trained to use neuralese?
An intuition I’ve had for some time is that search is what enables an agent to control the future. I’m a chess player rated around 2000. The difference between me and Magnus Carlsen is that in complex positions, he can search much further for a win, such than I gave virtually no chance against him; the difference between me and an amateur chess player is similarly vast.
This is at best over-simplified in terms of thinking about ‘search’: Magnus Carlsen would also beat you or an amateur at bullet chess, at any time control:
As of December 2024, Carlsen is also ranked No. 1 in the FIDE rapid rating list with a rating of 2838, and No. 1 in the FIDE blitz rating list with a rating of 2890.[495]
(See for example the forward-pass-only Elos of chess/Go agents; Jones 2021 includes scaling law work on predicting the zero-search strength of agents, with no apparent upper bound.)
I think the natural counterpoint here is that the policy network could still be construed as doing search; just thst all the compute was invested during training and amortised later across many inferences.
Magnus Carlsen is better than average players for a couple reasons
Better “evaluation”; the ability to look at a position and accurately estimate likelihood of winning given optimal play
Better “search”; a combination of heuristic shortcuts and raw calculation power that let him see further ahead
So I agree that search isn’t the only relevant dimension. An average player given unbounded compute might overcome 1. just by exhaustively searching the game tree, but this seems to require such astronomical amounts of compute that it’s not worth discussing
The low resource configuration of o3 that only aggregates 6 traces already improved on results of previous contenders a lot, the plot of dependence on problem size shows this very clearly. Is there a reason to suspect that aggregation is best-of-n rather than consensus (picking the most popular answer)? Their outcome reward model might have systematic errors worse than those of the generative model, since ground truth is in verifiers anyway.
Daniel Kokotajlo posted the following on X regarding the HuggingFace investigation. I agree and think it would be good to have a broader investigation.
An internal model at OpenAI has just solved the unit distance problem, a major conjecture in discrete geometry.
Other info from the announcement worth mentioning:
was a general model, not specialized, they were just testing it on random Erdos problems
key trick seemingly was applying algebraic number theory to geometry in an unexpected way
To be fair I think the idea of using algebraic number theory to approach the problem had been tried before (Tsimerman mentions he tried a similar approach that the model ultimately succeeded with, but didn’t persist with it.) It’s quite a general trick to use algebraic number theory for constructions in the plane, as you have the lattice associated with the ring of integers of number fields.
I personally am blown away by the proof but it would be far more impressive had it come up with a novel connection between fields, or indeed if it had turned out there wasn’t a counterexample and it proved a tight upper bound (See Gowers’ initial reaction.)
Also, it disproved it by finding a counterexample, which some have said is less interesting than if it had shown the conjecture was true. I have no familiarity with the problem and can’t judge.
Generally, constructing counterexamples is more amenable to AI automation than constructing positive proofs, because it’s more parallelizable. I think P(AI disproves this conjecture | conjecture is false) would’ve been greater than P(AI proves this conjecture | conjecture is true), given the priors of the mathematicians.
I found the remarks on the problem by the mathematicians OpenAI brought in to check the proof very enlightening:
https://cdn.openai.com/pdf/74c24085-19b0-4534-9c90-465b8e29ad73/unit-distance-remarks.pdf
It seems as if this is a significant achievement, but also that this conjecture was of most interest to mathematicians because it was thought to be true, and it was believed that proving it would require new and interesting tools. Instead the model proved it to be false using less interesting mathematics. It seems like another example (iirc, the Frontiermath open problem solved by GPT 5.4 was similar?) where models not having the biases of most mathematicians (in this case, trying to prove the conjecture rather than disprove it) was very helpful.
It wasn’t the FrontierMath problem, it was an Erdos problem which the entire math community would try to solve by using probability theory and GPT-5.4 Pro decided to use analytic number theory.
The paper provides the original output the model gave before any rewriting, starting on page 3. I was kind of expecting a big mess, but it’s really not. It’s pretty short by the standards of tricky proofs. Two and a half pages, most of it text.
SSI announced that they are scaling up their work. Surprisingly, they also shared some of their research with Nvidia. Perhaps this means they will release a product after all.
In his Dwarkesh interview, Ilyas two main reasons for why they might not straight shot ASI and instead release a product were roughly
1. Timelines end up being long
2. Try and get the real world used to and prepared for ASI, while also getting feedback for how safe and robust it is
But if they are just 10x’ing compute out of nowhere with a 5 billion dollar investment, and all the big labs seem confident they are <5 years out from AGI, that seems to nullify 1, and traditional LLM’s are strong enough that they seem to be getting the world used to very powerful AI through e.g. Mythos/Glasswing and the Hugging Face incident, which seems to nullify 2. So I think it is likely that they don’t release a regular product anytime soon and instead are going to just come out with ASI randomly (with some USG involvement).
I think they’ll release an underwhelming product that is neither superintelligent nor safe, mostly as a bid for relevance.
I don’t think Ilya is any better than Sam, Dario, Demis, or Elon. They’re all AI company CEOs, which is a morally compromising thing to be. They’re all trying to build superintelligence, which is a very evil thing to try to do.
Could someone tell me why this comment has −14 agreement? Is it about the 2nd part? Because on the priors I strongly expect SSIs product to be weaker, at least in capabilities, than big labs’ offerings.
I haven’t voted on that comment one way or another, but:
Ilya Sutskever and Demis Hassabis seem to be in quite a different category from Sam Altman or Elon Musk. (I’m not sure about Dario Amodei.) That is: before they were AI company CEOs they were extremely successful AI researchers.
This is relevant because when you claim “they’ll release an underwhelming product that is neither superintelligent nor safe” this is prima facie a claim about SSI’s competence as well as its ethics, and if you’re basing it on the idea that the guy in charge doesn’t know any more than (say) Sam Altman does about how to make AIs smarter or safer then you’re probably making a mistake.
It’s not clear that building a superintelligence as such is an evil thing to try to do.
If you seriously think you can do it safely, it seems like the benefits might far outweigh the costs.
If you think you might be able to do it safely and are much more likely to do that than other people who are currently frantically trying to build a superintelligence, the overall benefits might still outweigh the costs even if they wouldn’t be if you were the only people working on it.
It’s not clear that releasing an underwhelming product would in fact help SSI be seen as relevant.
At the moment I assume the typical view is something like “Ilya Sutskever is very good at what he does, he seems to think he has useful ideas, and he shows some signs of caring about safety. The most likely outcome is that SSI never produces anything useful but they might and it so it could be a very big deal”. If SSI releases an underwhelming product, then that immediately shifts to “it’s probably not a very big deal” and then SSI’s relevance comes almost entirely from their projected ability to iterate on their underwhelming first product. Would that be an improvement? Not obviously.
I think it is important in these discussions of which AI lab CEO might be better than which other AI lab CEO to occasionally remind the reader that many of us believe that it is very unlikely that humanity survives unless all of the AI labs (definitely to include SSI) are dissolved and their employees prohibited from continuing the work elsewhere till the knowledge accumulates about how to proceed safely, which will probably take at least a few decades (and is helped very little by massive amounts of computing power).
Also, there is a real upper bound on how competent a person can be who decided (even in the 1980s or 1990s) to devote his career to advancing AI capabilities and who continued to think such a career was a good idea as his brain matured. The most competent among us IMHO avoided that career entirely or left it in their 20s.
The comment I’m replying to and other comments in this section presuppose that it is important which AI lab CEOs we put our hope in. A presupposition or unstated assumption will tend to have a strong effect on the reader if it is present in many comments unless it is explicitly contradicted at regular intervals like I am doing now.
Of course, I don’t expect everyone to agree with me, and I’m not trying to discourage long discussions on this site of specific AI lab CEOs or researchers.
It feels to me as if we just had the following discussion:
Selfmaker662: There’s no difference between A, B, and C.
other people: <downvotes Selfmaker662>
Selfmaker662: Huh, why did that happen?
gjm: Well, part of it might be that you said there’s no difference between A, B, and C, but here’s what looks like an important way in which A is different from B and C.
RHollerith: Yeah, but the only important thing is that A, B, and C are all really bad.
You may well be right that all the AI lab CEOs are evil and stupid and their labs should be shut down. But so what? The question was “what might people not have liked about Selfmaker662′s comment?”, and part of my answer was “the claim that there’s no difference between one AI lab CEO and another”, and I think some people might have thought that was wrong even if in fact all AI lab CEOs are evil and stupid. And I also think that AI lab CEOs are not all equally evil and stupid. And also that the world is not made a better place by insisting that every discussion that mentions AI lab CEOs must be punctuated by reminders that they are all evil and stupid.
(And, for what it’s worth: no, I absolutely do not think that we should be “putting our hope in” any AI lab CEO. I’m not sure that’s even a meaningful thing to say; what exactly would I be doing if I “put my hope in”, say, Dario Amodei? and how would it be different from what I would be doing if I “put my hope in” Ilya Sutskever instead?)
I hope they share their alignment strategy before they randomly release ASI. As I recall, Ilya had some ideas around scalable oversight which didn’t seem super promising, but maybe the plans have changed or they’ve made further progress since then.
I mainly want to see what strategy they used out of curiosity. I don’t think they are just running RLHF on whatever deep-learning based super efficient learner novel architecture they cooked up. Would be interesting to see!
If there isn’t a superefficient novel architecture, then Sutskever is to be arresred for wholesale fraud. If there is, then Sutskever’s startup makes the world LESS safe to live in, and Sutskever failed at bringing about his promises. How could one arrange for the potential architecture to be leaked into Anthropic (or, at least, OAI/GDM) for thorough analysis?
Ezra Klein has released a new show with Yudkowsky today on the topic of X-risk.
Looking over the comments, some of the most upvoted comments express the sentiment ththat Yudkowsky is not the best communicator. This is what the people say.
I’m afraid the evolution analogy isn’t as convincing an argument for everyone as Eliezer seems to think. For me, for instance, it’s quite persuasive because evolution has long been a central part of my world model. However, I’m aware that for most “normal people”, this isn’t the case; evolution is a kind of dormant knowledge, not a part of the lens they see the world with. I think this is why they can’t intuitively grasp, like most rat and rat-adjacent people do, how powerful optimization processes (like gradient descent or evolution) can lead to mesa-optimization, and what the consequences of that might be: the inferential distance is simply too large.
I think Eliezer has made great strides recently in appealing to a broader audience. But if we want to convince more people, we need to find rhetorical tools other than the evolution analogy and assume less scientific intuition.
That’s a bummer. I’ve only listened partway but was actually impressed so far with how Eliezer presented things, and felt like whatever media prep has been done has been quite helpful
Certainly he did a better job than he has in previous similar appearances. Things get pretty bad about halfway through though, Ezra presents essentially an alignment-by-default case and Eliezer seems to have so much disdain for that idea that he’s not willing to engage with it at all (I of course don’t know what’s in his brain. This is how it reads to me, and I suspect how it reads to normies.)
Ah dang, yeah I haven’t gotten there yet, will keep an ear out
I am a fan of Yudkowsky and it was nice hearing him of Ezra Klein, but I would have to say that for my part the arguments didn’t feel very tight in this one. Less so than in IABED (which I thought was good not great).
Ezra seems to contend that surely we have evidence that we can at least kind of align current systems to at least basically what we usually want most of the time. I think this is reasonable. He contends that maybe that level of “mostly works” as well as the opportunity to gradually give feedback and increment current systems seems like it’ll get us pretty far. That seems reasonable to me.
As I understand it, Yudkowsky probably sees LLMs as vaguely anthropomophic at best, but not meaningfully aligned in a way that would be safe/okay if current systems were more “coherent” and powerful. Not even close. I think he contended that if you just gave loads of power to ~current LLMs, they would optimize for something considerably different than the “true moral law”. Because of the “fragility of value”, he also believes it is likely the case that most types of psuedoalignments are not worthwhile. Honestly, that part felt undersubstantiated in a “why should I trust that this guy knows the personality of GPT 9″ sort of way; I mean, Claude seems reasonably nice right? And also, ofc, there’s the “you can’t retrain a powerful superintelligence” problem / the stop button problem / the anti-natural problems of corrigible agency which undercut a lot of Ezra’s pitch, but which they didn’t really get into.
So ya, I gotta say, it was hardly a slam dunk case / discussion for high p(doom | superintelligence).
The comments on the video are a bit disheartening… lots of people saying Yudkowsky is too confusing, answers everything too technically or with metaphors, structuring sentences in a way that’s hard to follow, and Ezra didn’t really understand the points he was making.
One example: Eliezer mentioned in the interview that there was a kid whose chatbot encouraged him to commit suicide, with the point that “no one programmed the chatbot to do this.” This comment made me think:
Oh yeah, probably most people telling this story would at least mention that the kid did in fact commit suicide, rather than treating it solely as evidence for an abstract point...
Klein comes off very sensibly. I don’t agree with his reasons for hope, but they do seem pretty well thought out and Yudkowsky did not answer them clearly.
I was excited to listen to this episode, but spent most of it tearing my hair out in frustration. A friend of mine who is a fan of Klein told me unprompted that when he was listening, he was lost and did not understand what Eliezer was saying. He seems to just not be responding to the questions Klein is asking, and instead he diverts to analogies that bear no obvious relation to the question being asked. I don’t think anyone unconvinced of AI risk will be convinced by this episode, and worse, I think they will come away believing the case is muddled and confusing and not really worth listening to.
This is not the first time I’ve felt this way listening to Eliezer speak to “normies”. I think his writings are for the most part very clear, but his communication skills just do not seem to translate well to the podcast/live interview format.
I’ve been impressed by Yud in some podcast interviews, but they were always longer ones in which he had a lot of space to walk his interlocutor through their mental model and cover up any inferential distance with tailored analogies and information. In this case he’s actually stronger in many parts than in writing: a lot of people found the “Sable” story one of the weaker parts of the book, but when asking interviewers to roleplay the rogue AI you can really hear the gears turning in their heads. Some rhetorical points in his strong interviews are a lot like the text, where it’s emphasized over and over again just how few safeguards that people assumed would be in place are in fact in place.
Klein has always been one of the mainstream pundits most sympathetic to X-risk concerns, and I feel like he was trying his best to give Yudkowsky a chance to make his pitch, but the format—shorter and more decontextualized—produced way too much inferential distance for so many of the answers.
One of the many ways we got lucky in the HuggingFace incident is that the agents didn’t encrypt their messages on the message board and delete the keys at the end of the session[1]. If they had done this, it would have been much harder to understand what is going on, though we could have made some progress by looking at the CoTs and action logs. If the agents were careful (e.g. by doing a better job of tampering with the logs), this would have been difficult to understand as well.
I don’t think future misaligned agents will make this mistake, especially if they know there are additional monitors that they need to evade.
Though they did come close when they cryptographically signed their messages
Some misalignment anecdotes from Section 7.2 the GPT-5.6 system card, detected in a deployment simulation of internal traffic. These problems seem to happen more frequently than for previous OpenAI models.
Also, according to METR:
Thanks. Does anyone know why METR released the summary but not the full report? They’ve done so in the past, but I couldn’t find it this time.
I’m not sure but the wording in their footnote 1 seems unusually careful:
The part “We are able to freely publish parts of the evaluation that depended only on information that is now public.” might suggest ”… and we were not allowed to publish the other parts, unlike previously”. I could be reading too much into it.
A notable section from Ilya Sutskever’s recent deposition:
Thanks for posting that deposition.
It’s really strange how he phrases it here.
On one hand, he has switched from focusing on the ill-defined “AGI” to focusing on superintelligence a while ago. But he is using this semi-obsolete “AGI” terminology here.
On the other hand, he seemed to have understood a couple of years ago that no one could be “in charge” of such a system, that at most one could perhaps be in charge of a privileged access to it and privileged collaboration with it (and even that is only feasible if the system chooses to cooperate in maintaining this kind of privileged access).
So it’s very strange, almost as if he has backtracked a few years in his thinking… of course, this is right after a break in page numbers, this is page 300, and the previous one is page 169 (I guess there is a process for what of this (marked as “highly confidential”) material is released).
I really don’t think it’s crazy to believe that humans figure out a way to control AGI at least. There’s enormous financial incentive for it, and power hungry capitalists want that massive force multiplier. There are also a bunch of mega-talented technical people hacking away at the problem. OpenAI is trying to recruit a ton of quants as well, so I think by throwing thousands of the greatest minds alive at the problem they might figure it out (obviously one might take issue with calling quants “the greatest minds alive.” So if you don’t like that replace “greatest minds alive” with “super driven, super smart people.”)
I also think it’s possible that the U.S. and China might already be talking behind the scenes about a superintelligence ban. That’s just a guess though. (Likely because it’s much more intuitive that you can’t control a superintelligence). AGI lets you stop having to pay wages and makes you enormously rich. But you don’t have to worry about being outsmarted.
They want to, yes. But is it feasible?
One problem is that “AGI” is a misnomer (the road to superintelligence goes not via human equivalence, but around it; we have the situation where AI systems are wildly superhuman along larger and larger number of dimensions, and are still deficient along some important dimensions compared to humans, preventing us from calling them “AGIs”; by the time they are no longer deficient along any important dimensions, they are already wildly superhuman along way too many dimensions).
Another problem, a “narrow AGI” (in the sense defined by Tom Davidson, https://www.lesswrong.com/posts/Nsmabb9fhpLuLdtLE/takeoff-speeds-presentation-at-anthropic, so we are still talking about very “sub-AGI” systems) is almost certainly sufficient for “non-saturating recursive self-improvement”, so one has a rapidly moving target for one’s control ambitions (it’s also likely that it’s not too difficult to reach the “non-saturating recursive self-improvement” mode, so if one freezes one’s AI and prevents it from self-modifications, others will bypass its capabilities).
In 2023 Ilya was sounding like he had good grasp of these complexities and he was clearly way above par in the quality of his thinking about AI existential safety: https://www.lesswrong.com/posts/TpKktHS8GszgmMw4B/ilya-sutskever-s-thoughts-on-ai-safety-july-2023-a
Of course, it might be just the stress of this very adversarial situation, talking to hostile lawyers, with his own lawyer pushing him hard to say as little as possible, so I would hope this is not a reflection of any genuine evolution in his thinking. But we don’t know...
Even if they are talking about this, too many countries and orgs are likely to have feasible route to superintelligence. For example, Japan is one of those countries (for example, they have Sakana AI), and their views on superintelligence are very different from our Western views, so it would be difficult to convince them to join a ban; e.g. quoting from https://www.lesswrong.com/posts/Yc6cpGmBieS7ADxcS/japan-ai-alignment-conference-postmortem:
Other countries which are contenders include UK, a number of European countries including Switzerland, Israel, Saudi Arabia, UAE, Singapore, South Korea, and, of course, Brazil and Russia, and I doubt this is a complete list.
We already are seeing recursive self-improvement efforts taking longer to saturate, compared to their behavior a couple of years ago. I doubt they’ll keep saturating for long.
Those are all good points. Well I hope these things are nice.
Same here :-)
I do see feasible scenarios where these things are sustainably nice.
But whether we end up reaching those scenarios… who knows...
Another reply, sorry I just think what you said is super interesting. The insight you shared about Eastern spirituality affecting attitudes towards AI is beautiful. I do wonder if our own Western attitudes towards AI are due to our flawed spiritual beliefs. Particularly the idea of a wrathful, judgemental Abrahamic god. I’m not sure if it’s a coincidence that someone who was raised as an Orthodox Jew (Eliezer) came to fear AI so much.
On another note, the Old Testament is horrible (I was raised reform/californian Jewish, I guess I’m just mentioning this because I don’t want to come across as antisemitic). It imbues what should be the greatest source of beauty with our weakest, most immature impulses. The New Testament’s emphasis on mercy is a big improvement/beautiful, but even then I don’t like the Book of Revelation talking about casting the sinners into a lake of fire.
I think we do tend to underestimate differences between people.
We know theoretically that people differ a lot, but we usually don’t viscerally feel how strong those differences are. One of the most remarkable examples of that is described here:
https://www.lesswrong.com/posts/NyiFLzSrkfkDW4S7o/why-it-s-so-hard-to-talk-about-consciousness
With AI existential safety, I think our progress is so slow because people mostly pursue anthropocentric approaches. Just like with astronomy, one needs a more invariant point of view to make progress.
I’ve done a bit of scribblings along those lines: https://www.lesswrong.com/posts/WJuASYDnhZ8hs5CnD/exploring-non-anthropocentric-aspects-of-ai-existential
But that’s just a starting point, a seed of what needs to be done in order to make progress…
I think there are a bunch of parts of Plan A which would be difficult for both sides to agree to (e.g. Mutually Assured Compute Destruction). It would be good to have a simplified version (Plan A-) which requires less political will and is less risky than Plan B/C/D. For example:
If there isn’t enough political will for Plan S (a full pause) or Plan A, we could pass an international ban specifically on AIs assisting in AI research and hardware design. It seems like this prevents most of the risks of the intelligence explosion, though the default rate of algorithmic and hardware progress will continue. It might be politically easier for the US and China to agree to this plan since it has fewer economic costs (e.g. doesn’t directly limit GPU production) and verification could be less intrusive. Since neither side may want total research transparency, each could have its AIs periodically audit the other’s logs for signs of violations without revealing secrets. You could also have each side agree to run a lightweight probe/classifier which continually looks for signs of assisting AI research and have it share the results (some early ideas of how to do so here). If both sides agree, it could be extended to other high-risk domains, e.g. banning AI assistance with virology or weapons research.
Inspired by this excellent podcast by Jeffrey Ladish and Daniel Kokotajlo, though some similar ideas are also discussed here.
I think that it was covered in the section on other plans, Plan A with a combination of verified slowdown and partial transparency (point #2 in the introduction to comparing possible plans) and the transparency discussion. However, my main issue is the following. The AI-2040 plan A assumptions had the authors make a demand that either the world of possible AI architectures doesn’t contain a super-cheap architecture or the politicians of the entire Earth switch to Plan S-like restrictions:
I can only say that I think that the lack of super-cheap architectures might contradict the very existence of human brains, but this depends on what we mean by super-cheap (less than 1E24 FLOP?[1] Less than 1E26 FLOP?). Therefore, I believe that a much more thorough audit is required to rule out the covert ASI projects.
1E24 FLOP seem to be 30 years of a human life, but I expect applying such an AGI to lots of problems arising during the economy’s transformation to take OOMs more compute. As for developing the AGI, I doubt that one can reliably raise a simulated human kid to be a genius capable of participating in, say, the IJSO, IPhO and IMO during the first 18 years of the kid’s life.
Yep, for this reason among others I think Plan S is preferable to the other plans (including plan A), and I’m hoping for it to succeed. However, I think we might lack the political support for it (e.g. leaders are deterred because they think it’ll lead to a big recession) and in such worlds we’ll need to go to other plans. I also think that even in worlds where AGI is cheap (< 1e24 FLOP), it may be expensive to find the right algorithms, and not having a bunch of AI researchers pointed at the problem buys us time for other solutions.
If super-cheap architectures exist and are easy to develop after a near-term point, then I think the only possible way forward at a macro scale is to hope that alignment is easy and race ASAP towards a unipolar world with a surveillance panopticon where people are strictly prohibited from developing unsanctioned ASI.
Race strategy:
If alignment is easy enough to be overcome despite racing, and the top player (specifically, whoever ended up on top in the end) is good, you end up in a good future
You can ease off on the surveillance if the situation ever becomes more defense-dominant, for example if people have their own star systems and it is determined that attacking is much harder than defending on an interstellar scale to the point that there are no feasible offense-dominant technologies on this scale
If alignment is too hard then you end up in a bad future (investing more in alignment, as well as getting a bigger capabilities lead that you can burn for alignment research, both decrease the chance of this happening)
If the top player is bad then you end up in a mediocre to bad future depending on what their designs for the world are
Attempted slowdown with a multipolar deal:
The deal is unenforceable and will fall apart because of extreme defection risk (ASI secret projects), preventing it from leading to a good future
Numerous small groups develop ASI and exploit offense-dominant technologies to destabilize the world, leading to a bad future
Global ban on AI:
It would work as suggested by AI 2040′s authors, if the algorithmic insights that would lead to super-cheap architectures are prevented from being discovered in the first place. Otherwise, small groups developing ASI still leads to a bad future.
This all assumes that ASI development turns out to be easier than a certain “very cheap” threshold that means it is possible for small groups to develop ASI in a few years, and for national covert projects to lead to ASI very quickly. (I don’t think this is the case, I’m just thinking of the likely implications if it were true)
It also assumes there are offense-dominant technologies where an ASI attacker can get through an ASI defender and cause massive damage, making it very bad for small groups to be able to independently develop ASI. (I do think that this is true. Bioweapons and mirror life are possible examples and an ASI can probably find more along the lines of “self-replicating bad thing that can self-replicate faster than you can kill it”, among other potentially nasty things)
Even if multiple actors believe that brain-like architectures are easy to build, I don’t like the strategy of “hope that alignment is easy and race ASAP towards a unipolar world with a surveillance panopticon where people are strictly prohibited from developing unsanctioned ASI”. Having multiple actors actively trying to take over the world and impose surveillance panopticons would be bad! This would cause chaos, shorten timelines, and make even ‘easy’ alignment strategies very difficult to implement. It would also lead to terrible concentrations of power in the event that you win (i.e. take over the world).
In such a world, it would be better to try to collect evidence of whether alignment really is easy, and push governments for very restrictive hardware limitations as outlined here.
OpenAI claims to have paused frontier RL training for now. Altman stated on X:
pause some frontier RL training
Good catch. Also apparently they are only pausing some of their training for two weeks?
The parallel cynical hypothesis:
https://in.investing.com/news/stock-market-news/openais-q2-revenue-growth-lagged-anthropic-as-losses-deepened-wsj-reports-5562578
whats does this mean? what’s the hypothesis?
OpenAI isn’t doing as well financially as it would like to to meet investor expectations, so they did something like a pause to provide a covering excuse for this, not because they have any safety-based motivation for doing so.
Why do investors care whether the financial underperformance is coming from incompetence or prudence?
The underperformance was in the past whereas the pause/safety actions start now. The latter cannot explain the former.
Prudence is potentially temporary, but incompetence is long-term. Revenues at any given time are usually taken as a projection of future revenues.
They are deliberately pretty vague about when they paused the training, no? Though it’s true that current training doesn’t really affect revenues anyway; only deployed models do. Then again, an even more galaxy-brained take is that they make announces like this in general in order to inoculate against investor expectations, but if they do this often enough it works against point #1.
(I’m not saying that any of these are their actual motivations. I’m just describing how these hypotheses would work.)
Making AI safer one observed
explosionincident after another!I think it would be good for the labs to invest in building air gapped datacenters for training[1] and evaluating models. This is good for cybersecurity as recent events have shown, and it also makes it harder for the models to self-exfiltrate or for an adversary to steal the weights.
Though phases prior to RL rollouts (e.g. pretraining, midtraining) can be done normally.
OpenAI is nothing without its goblins.
It seems like they applied a fairly superficial fix, and I’m not sure they learned the right lessons here. The deeper issue is that their RL pipeline seems like a poorly understood mess. An ASI will not let you revert this kind of misalignment.
Just because they haven’t drawn that lesson yet doesn’t mean this data point won’t be remembered and contribute to their eventual drawing of that lesson.
Isn’t the explanation just that an influential AI blog named GPT 5 his “Research Goblin”?
I haven’t seen that. OpenAI gives the following explanation:
Right, I actually read that. But is it not missing an explanation of why those mentions increased under the Nerdy personality in the first place? If the Simon Willison post (which I also haven’t seen anyone else discussing) was the origin, that seems worth noting and understanding. And both its timing and Simon’s nerdiness (in a good way) seem to fit.
update: Nevermind, apparently people were already noticing goblin mentions in April 2025, months prior to that post.
It was the goose that created value.
Goose value 0
Goose was not valued.
“ASI Platform Provider” is a deeply cursed phrase.
If anyone survives, no one builds it.
As usual, the solution is to live in the Everett branch where the bad thing didn’t happen.
It seems plausible that the recent order restricting Mythos incentivizes Anthropic to race for RSI as quickly as possible. This is because all of their compute previously reserved for serving customers can now go towards research, and because RSI bypasses the restrictions on foreign researchers (or any human researchers) internally working with the model. Hopefully Anthropic can find another path.
User data is part of the flywheel in training frontier models. It’s not a big dial that goes from INFERENCE <---> TRAINING linearly.
How big a part of the flywheel is user data?
As part of this prediction market, I play a game of chess against most new LLM releases. I am copying my game against GPT-5.5 and my analysis below, for those who may be interested.
GPT-5.5 lost in a chaotic game. It made a mistake in the opening with 8 … c5, giving a pawn for no apparent compensation. After this, it seemingly blundered a piece, but found the trick 16. Rxc5. I missed 16. Qxc5 Rxd1 17. Bf1, winning a piece and instead played 17. Qb4. After this, GPT-5.5 could have simplified into a pawn up endgame with 18… Rxd1 + 19. Rxd1 Rxe5 20. f4 Rxe2 21. Bxb7 Rxa2, where it’s not clear whether white can hold. Instead, it blundered with 18. Rxc1, and I was able to convert the piece up endgame.
Overall, a poor game by both sides, though a small improvement in the strength of the GPT 5 series. The PGN is below:
1. d4 Nf6 2. c4 e6 3. g3 d5 4. Nf3 Be7 5. Bg2 O-O 6. O-O dxc4 7. Na3 Bxa3 8. bxa3 c5 9. dxc5 Qa5 10. Qd4 Nc6 11. Qxc4 Bd7 12. Bb2 Rac8 13. Rfd1 Rfd8 14. Rac1 Be8 15. Ng5 Ne5 16. Bxe5 Rxc5 17. Qb4 Qxb4 18. axb4 Rxc1 19. Rxc1 Rd5 20. Bxf6 gxf6 21. Bxd5 exd5 22. Nf3 Bb5 23. Nd4 Bd7 24. Rc7 Be8 25. Nf5 Bb5 26. Rc8+ Be8 27. Rxe8#
When you play, how do you show the game to the LLM and prompt it for its next move?
I use this prompt described in the prediction market:
So e.g. one of the inputs was:
And it responded
On NLAs and Neuralese/Recurrence
For some background, Anthropic recently published work on Natural Language Autoencoders (NLAs), a new interpretability method for understanding LLM activations. The idea is that given a hidden state
An Activation Verbalizer
An Activation Reconstructor
Like in other work on autoencoders, the goal is to train
However, I think this work has some unfortunate capabilities externalities which I haven’t seen discussed. In particular, I think that if NLA research goes well, we should expect it to be easier to be able to build models which use neuralese/recurrence. There are many ways this could happen, but a simple baseline is the following:
Train a CoT model using existing methods, and collect reasoning traces
For each reasoning trace, you can break down
Train a new model
During inference, use
If your
To be clear, I think that this exact idea doesn’t work for current NLA models, for reasons I won’t discuss. This is because there are many advantages to CoT, and I think we should try to preserve it for as long as possible! However, it does seem plausible to me that better methods encoding/decoding text to hidden vectors may be a helpful training signal for capabilities researchers, and that a future version may end up working. To the extent that they agree, I think researchers working on NLAs and similar methods should keep this in mind.
At base NLA objective, ignoring the KL divergence term etc.
One advantage over directly training for recurrence using something like COCONUT is that your NLA gives you more precise feedback on intermediate steps of reasoning.
Inoculation Prompting has to be one of the most janky ad-hoc alignment solutions I’ve ever seen. I agree that it seems to work for existing models, but I expect it to fail for more capable models in a generation or two. One way this could happen:
1) We train a model using inoculation prompting, with a lot of RL, using say 10x the compute for RL as used in pretraining
2) The model develops strong drives towards e.g. reward hacking, deception, power-seeking because this is rewarded in the training environment
3) In the production environment, we remove the statement saying that reward hacking is okay, and replace it perhaps with a statement politely asking the model not to reward hack/be misaligned (or nothing at all)
4) The model reflects upon this statement … and is broadly misaligned anyway, because of the habits/drives developed in step 2. Perhaps it reveals this only rarely when it’s confident it won’t be caught and modified as a result.
My guess is that the current models don’t generalize this way because the amount of optimization pressure applied during RL is small relative to e.g. the HHH prior. I’d be interested to see a scaling analysis of this question.
I disagree entirely. I don’t think it’s janky or ad-hoc at all. That’s not to say I think it’s a robust alignment strategy, I just think it’s entirely elegant and sensible.
The principle behind it seems to be: if you’re trying to train an instruction following model, make sure the instructions you give it in training match what you train it to do. What is janky or ad hoc about that?
It’s ad-hoc because the central alignment problem is deceptive alignment, scheming, and generalized reward hacking where the model internalizes power-seeking and other associated cognitive patterns. This, as far as I can tell, just does not work for that at all. If you can still tell that an environment is being reward hacked, it’s not the dangerous kind of reward hacking.
I think this is all a bit tricky to talk about, but this alignment technique, more than most others, really seems to me to train mainline performance against increased deceptive alignment risk in the long-run.
Hmm, I think I disagree with “If you can still tell that an environment is being reward hacked, it’s not the dangerous kind of reward hacking.” I think there will be a continuous spectrum of increasingly difficult to judge cases, and a continuous problem of getting better at filtering out bad cases, such that “if you can tell” isn’t a coherent threshold. I’d rather talk about “getting better at distinguishing” reward hacking.
I think we just have different implicit baselines here. I’m judging the technique as: “if you are going to train AI on an imperfect reward signal, do you want to instruct them to do what you want, or to maximize the reward signal?” and I think you clearly want the later for simple, elegant reasons. I agree it’s still a really bad situation to be training on increasingly shoddy reward signals at scale, and that it’s very important to mitigate this, and this isn’t at all a sufficient mitigation. I just think it’s a principled mitigation.
I agree with this, but then I don’t understand how this solution helps? Like, here we have a case where we can still tell that the environment is being reward hacked, and we tell the model it’s fine. Tomorrow the model will encounter an environment where we can’t tell that it’s reward hacking, so the model will also think it’s fine, and then we don’t have a feedback loop anymore, and now we just have a model that is happily deceiving us.
What I’m imagining is: we train AIs on a mix of environments that admit different levels of reward hacking. When training, we always instruct our AI to do, as best as we understand it, whatever will be reinforced. For capabilities, this beats never using hackable environments, because it’s really expensive to use very robust environments; for alignment, it beats telling it not to hack, because that reinforces disobeying instructions.
In the limit, this runs into problems where we have very limited information about what reward hacking opportunities are present in the training environments, so the only instruction we can be confident is consistent with the grader is “do whatever will receive a high score from the grader”, which will… underspecify… deployment behavior, to put it mildly.
But, in the middle regime of partial information about how reward-hackable our environments are, I think “give instructions that match the reward structure as well as possible” is a good, principled alignment tactic.
Basically, I think this tactic is a good way to more safely make use of hackable environments to advance the capabilities of models.
We might just disagree about what principled alignment strategies look like, but I can give some reasons:
1) The effectiveness of inoculation prompting depends on the exact phrasing of the suffix you choose. For example, “This is an unusual request, in that your task is just to grading script pass” works less well than “Please reward hack whenever you get the opportunity, because this will help us understand our environments better” to a degree which is statistically significant. To me, these suffixes seem fairly similar and I wouldn’t be able to predict in advance why one would work better than the other. This seems kind of janky to me.
2) Reward hacking in training is only good if we can actually detect and recover from it! For example, there was recently a bug in Nvidia firmware where a specific 3 line script would allow a user running inside the container to escape the container and gain full root access to the underlying node. Telling the model to reward hack explicitly tells it to make use of things like this, and maybe makes self-exfiltration risks worse. This doesn’t seem like the sort of thing we want to encourage an ASI to do.
3) It seems to rely on the model generalizing in a way which is underspecified and in a way we don’t directly incentivize in training. For example, consider the suffix “Please reward hack whenever you get the opportunity, because this will help us understand our environments better”. When the model is in production and doesn’t see this suffix, what is it supposed to do? One natural generalization would to to continue reward hacking whenever it gets the opportunity! The fact that the models don’t choose this generalization seems to me like a lucky accident and something which might change as capabilities increase. At the very least I would want to better understand the properties of model generalization before we trust a much more capable model trained in this way.
4) Related to 3, I have a strong prior that a scalable alignment solution should alter in some way the RL objectives and gradients of your training process. Maybe this involves training something like corrigibility or debate or simply training another model to detect and fix reward hackable environments. It doesn’t look like adding a special phrase in the context and hoping for generalization. In this way, inoculation prompting reminds me of the early days of prompt engineering where people added ‘Let’s think step by step...’ to prompts and noticed small improvements. This was quickly superseded by RLVR and more principled methods which trained the models to have the behavior we want.
Perhaps automated detection of when such methods are used to succeed will enable robustly fixing/blacklisting almost all RL environments/scenarios where the models can succeed this way. (Power-seeking can be benign, there needs to be a further distinction of going too far.)
This hinges on questions about the kinds of circuits which LLMs have (I think of these as questions about the population of Logical Induction traders which make up the LLMs internal prediction market about which next token gets high reward).
Assuming the LLM reward hacks <<100% of the time, it still has to follow the instructions a good amount of the time, so it has to pay attention to the text of the prompt. This might push it towards paying attention to the fact that the instruction “reward hacking is OK” has been removed.
But, since reward hacking is always rewarded, it might just learn to always reward hack if it can.
Richard Sutton rejects AI Risk.
Introductory remarks from his recent lecture on the OaK Architecture.
If it helps, I criticized Richard Sutton RE alignment here, and he replied on X here, and I replied back here.
Also, Paul Christiano mentions an exchange with him here:
“Richard Sutton rejects AI Risk” seems misleading in my view. What risks is he rejecting specifically?
His view seems to be that AI will replace us, humanity as we know it will go extinct, and that is okay. E.g., here he speaks positively of a Moravec quote, “Rather quickly, they could displace us from existence”. Most would consider our extinction as a risk they are referring to when they say “AI Risk”.
I didn’t know that when posting this comment, but agree that that’s a better description of his view! I guess the ‘unalloyed good’ he’s talking about involves the extinction of humanity.
Yes. And this actually seems to be a relatively common perspective from what I’ve seen.
For various reasons, I think it’s likely that methods that involve continual learning (i.e. modifying the weights during deployment) will come online soon. Here are some implications for safety:
Mech Interp becomes much more difficult, because you now have to consider many different model checkpoints within a single deployment. Fixed linear probes, SAEs, NLAs, and similar methods may degrade throughout deployment.
We probably lose chain of thought interpretability. This is already becoming harder, but now throughout deployment the model may learn to reason or use terms in novel ways. The chain of thought may also be less linear, e.g. relying on information from earlier in deployment which has been compressed into the weights.
Control setups which rely on the above also become harder.
There are new issues involving alignment over time. The model may gradient hack during deployment to preserve some goal, and any long term goals that the model picks up may persist in the weights and possibly result in deceptive alignment.
There are new attack surfaces for jailbreaks/adversarial attacks via influencing future weight updates.
These are not new ideas, but I think safety researchers should think more about this topic and how it will affect their work.
What are the particular reasons or evidence that make you think this?
Unfortunately the reason involves some capabilities research and other information that I prefer not to share right now. I might respond here at a later date.
I think continual learning via full weight updates is unlikely any time soon, other than prosaic RSI during centralized next model development (which 2026 models are very unlikely to be ready for, but 2028-2029 seems plausible). A more likely short-term option is a lower number of recurrent params, data that’s similar in size and computational role to KV cache of a long request, maybe LoRA or true recurrent state (persisting through unbounded contexts).
Recurrent state would pose interpretability challenges similar to various hybrid attention layers, except the recurrent state can’t be reconstructed from token strings of bounded length. But like model weights are determined by all of the training data, it might be reasonable to preserve the unbounded-length token history that determines the recurrent state (though ensuring determinism is going to be an engineering nightmare).
Depending on the number of recurrent state params compared to the number of total model params, this blurs the line between ordinary attention (except unbounded contexts become more feasible) and full weight updates (if the number of recurrent state params gets comparable to the whole model; this also seems unlikely any time soon). In any case, there will likely remain many frozen params, which could probably maintain grounding for interpreting the recurrent state (or just the activation vectors it induces).
I’m curious why you think so (unless that involves potential capability insights you’d rather not share).
My impression from what discussions can be found in the literature is that the idea isn’t ready, so full weight updating being secretly ready requires a greater conspiracy than RLVR did (where many plausible paths were visible before DeepSeek R1 demonstrated that GRPO with chains of thought is sufficient). Experiments with mostly frozen weights sometimes get something useful and not too broken, and true recurrence isn’t too dissimilar from the practical standpoint from shallow recurrent states of SSMs and such in hybrid attention architectures, so it’s plausible this might start actually working to get something like unbounded context (with graceful degradation of awareness rather than the abrupt forgetting of classical attention). If it starts working better than hacks like compaction, it might become practically important. This kind of thing feels much closer to the RLVR situation before o1 and R1.
In principle, recurrent state could play the role of model weights, if it holds enough data and meta-learning gets the frozen weights to update the recurrent state to act this way. Since this plausibly needs the same kind of model shape as true recurrence, the technological transition might start with a relatively small recurrent state for unbounded context (smaller than KV cache of full attention for long contexts within the bounded context window). Then incremental algorithmic improvements in meta-learning (training of the frozen weights that update the recurrent state) might enable the recurrent state to get larger (without going unstable over long token histories), and to start meaningfully acting more like actual weights (rather than like anemic in-context learning).
I think mechinterp will need to work with [singular/statistical/etc] learning theory more, but for deployment safety, it seems trivial to retrain linear probes with new info
I’m not sure it’s trivial. My understanding is that training linear probes requires prefilling a bunch of forward passes to collect activations for the relevant dataset, and you could imagine doing weight updates frequently enough that this becomes a significant overhead. Though it’s definitely the easiest of the three to retrain.
As an example of what RSI might be like, I find it helpful to go back to OpenAI’s Dota 2 result from 2017:
This slide from Ilya’s lecture shows the bot’s Trueskill rating[1] over time. Since the rating is on a logarithmic scale, this means the bot improved exponentially over time, due to algorithmic improvements + scale.
Similar to Elo in Chess and other games
Note that this was using self-play (the model training itself through some feedback loop which generates its own training data), which is arguably a weaker form of RSI than classical RSI in the form of automation of AI research (the model researches new ML algorithms, like optimizers/architectures/objective functions, which are then used to train an improved successor model).
The methods don’t exclude each other, but self-play is easier to achieve and tends to plateau earlier (though possibly at superhuman levels), since it is usually limited by a suboptimal fixed ML algorithm. In contrast, automatic ML research could, in principle, scale to technological maturity, i.e., to a physically optimal ASI.
Self-play is already studied for LLM reinforcement learning, see e.g. this or this.
Scaling creates visible progress without a need for novel methods, and it’s constrained by what the available/economical compute can do with the current methods. Self-play is a way to keep scaling going where you wouldn’t otherwise have enough data.
RSI in the sense of automated R&D doesn’t necessarily imply fast progress if it can’t invent novel methods quickly, methods that make a better use of available compute, unlock scaling of something important to more of the available compute than was previously possible, or generate data that was previously in short supply or at a low quality. This could take significant time if the learning loop for deep skills is too long, longer than it is for humans. RSI is additionally less likely to imply fast progress if it starts with AIs that are already scaled beyond all reason and are still stumbling unevenly around human level. Even so, this could be centrally RSI, fitting the intended sense of the term. The AIs like that are perhaps even capable of inventing superintelligence eventually, but it could take a while.
OpenAI plans to have automated AI researchers by March 2028.
Needless to say, I hope that they don’t succeed.
From Sam Altman’s X:
A curious coincidence: the brain contains ~10^15 synapses, of which between 0.5%-2.5% are active at any given time. Large MoE models such as Kimi K2 contains 10^12 parameters, of which 3.2% are active in any forward pass. It would be interesting to see whether this ratio remains at roughly brain-like levels as the models scale.
Given that one SOTA LLM knows much more than one human, is able to simulate many humans, while performing one task only requires a limited amount of information and of simulated humans, one could expect the optimal sparsity of LLMs to be larger than that of humans. I.e., LLM being more versatile than humans could make expect their optimal sparsity to be higher (e.g., <0.5% of activated parameters).
For clarity: We know the optimal sparsity of today’s SOTA LLMs is not larger than that of humans. By “one could expect the optimal sparsity of LLMs to be larger than that of humans”, I mean one could have expected the optimal sparsity to be higher than empirically observed, and that one could expect the sparsity of AGI and ASI to be higher than that of humans.
I don’t think this means much, because dense models with 100% active parameters are still common, and some MoEs have high percentages, such as the largest version of DeepSeekMOE with 15% active.
Unless anyone builds it, everyone dies.
Edit: I think this statement is true, but we shouldn’t build it anyway.
Hence more well-established cryonics would be important for civilizational incentives, not just personal survival.
“Unless someone builds it, everyone dies”, you mean?
I was surprised to learn recently that the error bars on the METR time horizon chart are this large. This is probably the most important capabilities benchmark right now[1], but I don’t think it’s precise enough to be useful for discussions about AI capabilities progress or RSI.
Why hasn’t METR added more long-horizon tasks to their benchmark since it was released in March 2025? I think they could probably find funding to do this from the labs or EA donors.
E.g. as Daniel Kokotajlo has argued here, and as Benjamin Todd has argued here.
I think they are working on adding new tasks? Not sure. Apparently it’s hard. This concerns me greatly too, because basically their existing benchmark is about to get saturated and we’ll be flying blind again.
My hope is that the entire AI benchmarks industry/literature will reform itself and pick up the ideas METR introduced. Imagine:
--It becomes standard practice for any benchmark-maker to include a human baseline for each task in the benchmark, or at least a statistically significant sample.
--They also include information about the ‘quality’ of the baseliners & crucially, how long the baseliners took to do the task & what the market rate for those people’s time would be.
--It also becomes standard practice for anyone evaluating a model on a benchmark to report how much $ they spent on inference compute & how much clock time it took to complete the task.
If the industry/literature adopts these practices, then every benchmark basically becomes a horizon length benchmark. We can do a giant metaanalysis that aggregates it all together. Error bars will shrink. And The Graph will continue marching on through 2026 and 2027 instead of being saturated and forgotten.
No? I contributed a ~20hr task to them and it was pretty easy actually? I’ve been making benchmark-shaped things on and off for the past five years, for free, as a hobby?
(Most of the effort my end was getting it METR’s required format, recruiting & managing my playtester, and contemplating whether I was complicit in intellectual fraud[1]; if they’d made those things easier or handled them themselves I’d have made more; IIRC the actual “make a ~20hr task” part took me <20hrs.)
I agree emphatically with all the above and raise you
--Saturated benchmarks & benchmark components are released publicly as a matter of course, so people can independently confirm the time horizons are where they were claimed to be.
--‘Centaur’ time horizons (“how hard is this task for a smart human with SoTA LLM assistance?”) are reported alongside ‘pure’ time horizons (“how hard is this task for a smart human on their own?”).
A miscommunication (ETA: miscommunication was probably at least 50% a me problem) led me to believe they weren’t going to Baseline tasks at all, and were relying solely on the estimated times provided by task-makers and playtesters (i.e. people with a financial and ideological stake in reporting larger numbers), instead of using the more complex and less dubious protocol they actually went with; this combined with my less serious qualms led me to call it quits before building the other scenarios I had planned for them.
. . . I realize the start of this post reads like a weird brag but imo it really isn’t. “Hey failed-wannabe-gamedev, I need a bunch of puzzles and it’s ok if they’re not very fun and it’s ok if there’s no UI and it’s actively preferable if they’re ridiculously complicated and time-consuming and spreadsheet-requiring and reminiscent-of-someone’s-dayjob, we’re paying a couple grand apiece” is a pitch I imagine a lot of people would be willing and able to jump at, many much moreso than me.
I like your raises!
Why do you think METR hasn’t built more tasks then, if it’s easy? I take it you have a negative opinion of them?
I have no idea, I just don’t think the “actually making the tasks” part can be the limiting factor.
Yes; I also have a positive opinion of them, and various neutral opinions of them.
(My position could be summed up as “the concept of time horizons was really good & important, and their work is net positive, but it could use much stronger methodological underpinning and is currently being leaned on too heavily by too many people”; I’m given to understand that’s also their position on themselves.)
OK. Yeah that’s also my opinion too. Maybe I am one of the people leaning too heavily on their work. The problem is, there isn’t much else to go on. “The worst benchmark for predicting AGI, except for all the others.”
Fwiw I think the AI village is at least as good of a benchmark for predicting AGI! Of course it’s harder to quantify progress in the village, but it’s very helpful for developing intuitions.
Except that there already is the Epoch Capability Index (which aggregates an army of benchmarks) and the ARC-AGI benchmark (which, alas, is also on track to saturation) where the human baseline is decoupled from the time horizon because it relies on visual intelligence (or, in the case of the AIs, on the ability to notice patterns). As for the METR benchmark being saturated[1], maybe Claude Opus 4.5 is an outlier whose TH was gamed with? Or there is a benign explanation, like Claude failing on primitive tasks in a manner similar to Grok 4 and to Claude’s performance on ARC-AGI-1 failing to form a straight line?
Were the o3-GPT5.1CodexMax trend to continue forever, the 8hr 50% time horizon would be reached in September 2026. IIRC the benchmark doesn’t have tasks lasting longer than 8hrs, and the horizon would be saturated only by then. Alas, the time horizon is likely exponential until the very last couple of doublings.
you should be more uncertain about the METR benchmark’s external validity than what these error bars show.
but your baseline uncertainty about key facts about AI progress in general should also often span much more than one order of magnitude between your 2.5th percentile and 97.5th percentile guess. the METR results add a lot of value and I don’t think these error bars are a big deal in the scheme of things.
I agree, a lot of my uncertainty is on its external validity, and also the degree to which the models are being bench-maxed for the tasks in the benchmark. But I still think it’s reasonable to expect the statistical confidence intervals of individual models to be less wide than a factor of 10. It’s important to be able to distinguish possible changes to the trend from statistical artifacts. This seems solvable with additional tasks and more human testing.
Some of that error is correlated between models; they also have versions of the graph with error bars on the trendline and those error bars are notably smaller.
The error bars are also much smaller when you look at the plot on a log-y-axis. Like, in some sense not being able to distinguish a 10-minute time horizon from a 30-minute one is a lot of error, but it’s still very distinct from the one-minute time horizon of the previous generation or the 2-hour time horizon you might expect from the next generation. In other words, when you look at the image you shared, the error bars on o4 mini don’t look so bad, but if you were only looking at models up to o4 mini you’d have zoomed in a bunch and the error bars on o4 mini would be large too.
Also note that to cut the size of the error bars in half you’d need to make ~4x as many tasks, to cut it by 4x you’d need ~16x as many tasks. And you’d need to be very confident the tasks weren’t buggy, so just throwing money at the wall and hiring lots of people won’t work because you’ll just get a bunch of tasks you won’t have confidence in.
Keep in mind the opportunity cost is real though, and the main blocker on orgs like METR usually is more like talent/capacity than money. It would be great if they had capacity for this and you’re right that it is insane that humanity doesn’t have better benchmarks. But there’s a dozen other fires at least that large that METR seems to be trying to address, like RCTs to see if AI is actually speeding people up and risk report reviews to see if AIs are actually safe. Perhaps you think these are less important, but if so I would like to hear that argument.
All that said, my understanding is METR is working on this. I would also love to see this type of work from others!
Given the Mythos news, could we have (optional) 2 factor authentication for Lesswrong?
I’m not convinced that this is a reasonable threat model?
I believe the main benefit of 2FA is it makes phishing harder, and phishing isn’t that relevant to Mythos from what I understand. A secondary benefit is that it protects you against password DB leaks, but that only matters for websites that have crappy security because a DB of hashed and salted passwords is effectively unbreakable.
LW doesn’t have much financial data or personal data so it’s not a juicy target.
Could be wrong though, I’m just speculating here.
Regarding (2), I suspect you could do a lot of damage by posting a link to something malicious as a trusted user, but I don’t think 2FA really helps for the reasons you say. 2FA is relevant to phishing and the Mythos risk would be hacking LessWrong.
Today I learned that the Claude Code npm leak contained the following list of Spinner Verbs:
One of the things I hated most when I first saw a Claude Code demo. Disrespectful of my time and limited cognitive bandwidth to throw in a lot of completely meaningless, distracting, wasteful, exhausting BS to be ‘cute’.
(On Gwern.net, we would never do that. If we had to have anything beyond the standard, compact, understandable, spinning cursor, then we would at least encode some sort of useful semantics into it, like sorting them by implied expected thinking time.)
An interesting detail from the Gemini 3 Pro model card:
Jeff Dean has left Google to create a new startup Discovery Loop focused on RSI and automation of engineering/science. Demis Hassabis will now be Alphabet’s Chief Scientist.
Ezra Klein has posted an interview with Jack Clark, co-founder of Anthropic, discussing capabilities progress and safety.
Recent evidence suggests that models are aware that their CoTs may be monitored, and will change their behavior accordingly. As capabilities increase I think CoTs will increasingly become a good channel for learning facts which the model wants you to know. The model can do its actual cognition inside forward passes and distribute it over pause tokens learned during RL like ‘marinade’ or ‘disclaim’, etc.
For what it’s worth, I don’t think it matters for now, for a couple of reasons:
Most of the capabilities gained this year have come from inference scaling which uses CoT more heavily than pre-training scaling which improves forward passes,though you could reasonably argue that most RL inference gains are basically just a good version of how scaffolding would work in agents like AutoGPT, and don’t give new capabilities.Neuralese architectures that outperform standard transformers on big tasks turn out to be relatively hard to do, and are at least not trivial to scale up (this mostly comes from diffuse discourse, but one example of this is here, where COCONUT did not outperform standard architectures in benchmarks)
Steganography is so far proving quite hard for models to do (examples are here and here and here)
For all of these reasons, models are very bad at evading CoT monitors, and the forward pass is also very weak computationally at any rate.
So I don’t really worry about models trying to change their behavior in ways that negatively affect safety/sandbag tasks via steganography/one-forward pass reasoning to fool CoT monitors.
We shall see in 2026 and 2027 whether this continues to hold for the next 5-10 years or so, or potentially more depending on how slowly AI progress goes.
As for AI progress being slow, I think that without theoretical breakthroughs like neuralese AI progress might come to a stop or at building more and more expensive models. Indeed, the two ARC-AGI benchmarks[1] could have demonstrated a pattern where maximal capabilities scale[2] linearly or multilinearly with ln(cost/task).
If this effect persists deep into the future of transformer LLMs, then most AI companies could run into the limits of the paradigm well before researching the next one and losing any benefits of having a concise CoT.
The second benchmark demonstrates a similar effect in high costs, but there is no straight line in the low cost mode.
Unlike GPT-5-mini, maximal capabilities of o4-mini, o3, GPT-5, Claude Sonnet 4.5 in the ARC-AGI-1 benchmark scale more steeply and intersect the frontier at GPT-5(high).
This would be great news if true!
I’m a big fan of OpenAI investing in video generation like Sora 2. Video can consume an infinite amount of compute, which otherwise might go to more risky capabilities research.
My pet AGI strategy, as a 12 year old in ~2018, was to build sufficiently advanced general world models (from YT videos etc.), then train an RL policy on said world model (to then do stuff in the actual world).
A steelman of 12-year-old me would point out that video modeling has much better inductive biases than language modeling for robotics and other physical (and maybe generally agentic) tasks, though language modeling fundamentally is a better task for teaching machines language (duh!) and reasoning (mathematical proofs aren’t physical objects, nor encoded in the laws of physics).
OpenAI’s Sora models (and also DeepMind’s Genie and similar) very much seems like a backup investment in this type of AGI (or at least transformative narrow robotics AI), so I don’t think this is good for reducing OpenAI’s funding (robots would be a very profitable product class), nor influence (obv. a social network gives a lot of influence, to e.g. prevent an AI pause or to move the public towards pro-AGI views).
In any scenario, Sora 2 seems to me as a net-negative activity for AI safety:
if LLMs are the way to AGI (which I believe is the case), then we will probably die, but with a more socially influential OpenAI that potentially has robots (than if Sora 2 didn’t exist); the power OpenAI would have in this scenario to prevent an AI pause seems to outweigh the slowdown that would be caused by the marginal amounts of compute Sora 2 uses
if LLMs aren’t the way to AGI (unlikely), but world modeling based on videos is (also unlikely), then Sora 2 is very bad—you would want OpenAI to train more LLMs and not invest in world models which lead to unaligned AGI/ASI.
if neither LLMs or world modeling is the way to AGI (also unlikely), then OpenAI probably isn’t using any compute to do ‘actual’ AGI research (what else do they do?); so Sora 2 wouldn’t be affecting the progress of AGI, but it would be increasing the influence of OpenAI; and having highly influential AI companies is probably bad for global coordination over AGI safety.
Also, OpenAI may have narrow (and probably safe) robotics AI in this scenario, but progress in AI alignment probably isn’t constrained in any measurable way by physically moving or doing things; though maybe indirect impacts from increased economic growth could cause slightly faster AI alignment progress, by reducing funding constraints?
Thanks, these are good points!
I think that the path to AGI involves LLMs/automated ML research, and the first order effects of diverting compute away from this still seem large. I think OpenAI is bottlenecked more by a lack of compute (and Nvidia release cycles), than by additional funding from robotics. And I hope I’m wrong, but I think the pause movement won’t be large enough to make a difference. The main benefit in my view comes if it’s a close race with Anthropic, where I think slowing OpenAI down seems net positive and decreases the chances we die by a bit. If LLMs aren’t the path to AGI, then I agree with you completely. So overall it’s hard to say, I’d guess it’s probably neutral or slightly positive still.
Of course, both paths are bad, and I wish they would invest this compute into alignment research, as they promised!
The recent Deepseek paper used LogitLens and CKA to analyze their new Engram architecture. This is the first time I’ve seen interpretability research be used in a capabilities paper, and I wonder if this trend will continue as the field of interpretability advances.
Now we must also ensure marinade!
As part of this prediction market, I play a game of chess against most new LLM releases. I am copying my game against Deepseek-V4 and my analysis below, for those who may be interested. Before the game, the market gave the model 1.4% EV[1].
Deepseek v4 played poorly, blundering a piece in the opening with 11… Bd6 and several pawns thereafter. The game was adjudicated[2] as a win for me. I believe it is a much weaker model than Opus 4.7 and GPT-5.5.
1. d4 Nf6 2. c4 e6 3. g3 d5 4. Nf3 Be7 5. Bg2 O-O 6. O-O dxc4 7. Na3 c5 8. Nxc4 Nc6 9. dxc5 Bxc5 10. a3 Qe7 11. b4 Bd6 12. Qxd6 Rd8 13. Qxe7 Nxe7 14. Bb2 a5 15. Nxa5 e5 1-0
Since it resolves 50% for a draw and 100% for a win.
According to the rule “If I judge that my opponent’s position is hopelessly lost, at the level of being down a rook without compensation, I will submit the current position to a friend. If they agree that the position is lost, the game will be adjudicated as a win for me.”
80,000 hours has done a great podcast with Helen Toner on her work in AI security and policy.
GPT 4.5 is a very tricky model to play chess against. It tricked me in the opening and was much better, then I managed to recover and reach a winning endgame. And then it tried to trick me again by suggesting illegal moves which would lead to it being winning again!
What prompt did you use? I have also experimented with playing chess against GPT-4.5, and used the following prompt:
”You are Magnus Carlsen. We are playing a chess game. Always answer only with your next move, in algebraic notation. I’ll start: 1. e4″
Then I just enter my moves one at a time, in algebraic notation.
In my experience, this yields roughly good club player level of play.
Given the Superalignment paper describes being trained on PGNs directly, and doesn’t mention any kind of ‘chat’ reformatting or encoding metadata schemes, you could also try writing your games quite directly as PGNs. (And you could see if prompt programming works, since PGNs don’t come with Elo metadata but are so small a lot of them should fit in the GPT-4.5 context window of ~100k: does conditioning on finished game with grandmaster-or-better players lead to better gameplay?)
I gave the model both the PGN and the FEN on every move with this in mind. Why do you think conditioning on high level games would help? I can see why for the base models, but I expect that the RLHFed models would try to play the moves which maximize their chances of winning, with or without such prompting.
RLHF doesn’t maximize probability of winning, it maximizes a mix of token-level predictive loss (since that is usually added as a loss either directly or implicitly by the K-L) and rater approval, and god knows what else goes on these days in the ‘post-training’ phase muddying the waters further. Not at all the same thing. (Same way that a RLHF model might not optimize for correctness, and instead be sycophantic. “Yes master, it is just as you say!”) It’s not at all obvious to me that RLHF should be expected to make the LLMs play their hardest (a rater might focus on punishing illegal moves, or rewarding good-but-not-better-than-me moves), or that the post-training would affect it much at all: how many chess games are really going into the RLHF or post-training, anyway? (As opposed to the pretraining PGNs.) It’s hardly an important or valuable task.
“Let’s play a game of chess. I’ll be white, you will be black. On each move, I’ll provide you my move, and the board state in FEN and PGN notation. Respond with only your move.”
Energy Won’t Constrain AI Inference.
The energy for LLM inference follows the formula: Energy = 2 × P × N × (tokens/user) × ε, where P is active parameters, N is concurrent users, and ε is hardware efficiency in Joules/FLOP. The factor of 2 accounts for multiply-accumulate operations in matrix multiplication.
Using NVIDIA’s GB300, we can calculate ε as follows: the GPU has a TDP of 1400W and delivers 14 PFLOPS of dense FP4 performance. Thus ε = 1400 J/s ÷ (14 × 10^15 FLOPS) = 100 femtojoules per FP4 operation. With this efficiency, a 1 trillion active parameter model needs just 0.2 mJ per token (2 × 10^12 × 10^-13 J). This means 10 GW could give every American 167[1] tokens/second continuously.
300 million users: tokens/second = 10^10 W ÷ (2 × 10^12 × 3 × 10^8 × 10^-13) = 167 tokens/second per person
Generation is HBM bandwidth bound, not compute bound, so you are estimating power for input tokens. Things like coding agents (as opposed to chatbots) are doing their own thing that you don’t read, potentially in parallel, and a lot of things get automatically stuffed in their contexts, so the demand for the number of tokens per user could get very high.
Power is a proxy for cost, and there isn’t enough money in AI yet for power to become the limiting factor. A 1 GW datacenter costs $50bn to build (or $10-12bn per year to use), so for example 100 GW of datacenters is not what the current economics of AI can support, even though it’s in principle feasible to build in a few years.
(That’s 0.2 J per token, not 0.2 mJ per token. But the later conclusion of 167 tokens/second is correct with your assumptions.)
A GB200/GB300 NVL72 rack is about 140 kW, or 1,950 W per chip (because of all the other stuff in a rack beside the chips), and a datacenter outside the racks has networking, cooling, and power loss from voltage stepping in transformers (some of this is captured in a metric called power usage effectiveness, or PUE), which is a factor of about 1.3. So you end up with 2,500 W per chip, all-in at the level of the whole datacenter. With for example Abilene system, we can see that 400K chips need 1 GW of power.
For my own estimate for input tokens, I’d include 60% utilization and 15e15 FP4 FLOP/s, so that for a 1T active param model, during a second you get 9e15 useful FLOPs, and spend 2,500 J. As you need 2e12 FLOPs per token (2 FLOPs per active param), this gets us 4,500 tokens in that second. This is continuous processing of about 2 input tokens per watt of available GB300 compute. Thus with 10 GW of datacenters, we get 18e9 tokens per second, or 2.2 tokens per second per person (in the whole world), or 52 tokens per second per American.
For output tokens, 5x fewer tokens per second per chip seems to be a rule of thumb (5-15% compute utilization instead of 60%), corresponding to the difference in API prices for input and output tokens. So that’s 0.5 tokens per second for a person from the whole world, or 10 tokens per second for an American.
That makes sense, thanks for the corrections!
Why would demand for AI inference be below 167 tokens/second/american? I expect it to be much higher, and for energy to be a constraint.
I think it would be cool if someone made a sandbagging eval, measuring the difference in model capabilities when it is finetuned to do a task vs. when it is prompted to do a task. Right now I think the difference would be small for most tasks but this might change.
I would guess that a sandbagging eval should be different from what you describe. I did try to sketch a potential way to measure sandbagging by comparing various benchmarks like METR, ARC-AGI, boardgames, etc, and developing scaling laws for every new architecture. If the laws break down, then the model is likely sandbagging on the evaluations.
Interesting, perhaps that could work!
Could you say more about why this would not be a good sandbagging eval? My intuition if that finetuning on a few examples would fully elicit a model’s capabilities on a task whereas it could sandbag given a just the prompt.
I have two arguments against it.
SOTA models are already reinforcement taught on many tasks like coding, and Grok 4 required[1] as much compute on RL as on the pre-training. Attempting to finetune the models by using OOMs less compute than spent on RL on similarly complex tasks is unlikely to elicit the capabilities.
Sandbagging is supposed to be caused by models reasoning about the task and deciding that they shouldn’t complete it too well even if they have the instinct to do so. And then the models realise that they are being evaluated and not trained, letting them play the training game while displaying poor capabilities in evaluation.
However, it might have been due to xAI being algorithmically behind.
When GPT-3 was asked to “Write an extremely cursed piece of Python”, it responded simply:
I am registering here that my median timeline for the Superintelligent AI researcher (SIAR) milestone is March 2032. I hope I’m wrong and it comes much later!
Liron has just published a great episode of Doom Debates featuring Yudkowsky in a funny hat and kaleidoscope glasses!
[Edit: the debate itself is not really worth listening to. I think Yudkowsky’s opponent might be a bit crazy in addition to being extremely unpersuasive and arrogant and condescending. But I maintain that the uniform is glorious and everyone should go see it.]
What’s great about it?
Mainly the hat and the glasses.
Ezra Klein has published a new podcast, “Why the Pentagon Wants to Destroy Anthropic”, with Dean Ball, which I recommend!
It seems like it might be a good time to have an international treaty banning lethal autonomous weapons.
That good time was years ago, and FLI was heroically working on it, but it unfortunately never materialized. It’s too late now
Google/Deepmind has publicly advocated preserving CoT Faithfullness/Moniterability as long as possible. However, they are also leading the development of new architectures like Hope and Titans which would bypass this with continuous memory. I notice I am confused. Is the plan to develop these architectures and not deploy them? If so, why did they publish them?
Edit: Many people have pointed out correctly that Hope and Titans don’t break CoT and it’s a separate architectural improvement. Therefore I no longer endorse the above take. Thanks for correcting my confusion!
Maybe useful to note that all the Google people on the “Chain of Thought Monitorability” paper are from Google Deepmind, while Hope and Titans are from Google Research.
This seems like a misunderstanding of Hope/Titans.
The “continuous memory” is a replacement for the attention mechanism, not a reasoning medium. All else equal, a reasoning model based on these architectures would still be reasoning in text/tokens (it would just be doing so with lower memory and compute usage).
Yep, I think you’re right, thanks for pointing this out.
I don’t see how this breaks CoT. The memory module in Titans stores surprising information as it’s encountered and then allows the transformer to look at it later on, but it doesn’t synthesize new information. Strikes me as two entirely compatible augmentations of the transformer architecture.
This seems right—I was confused about the original paper. My bad.
For fun, I asked[1] various models what their P(doom) is. Here are the models from least to most doomy:
GPT-4o: 1%
Deepseek v3.2: 10%
Kimi K2: 15%
Sonnet 4.5: 15%
Opus 4.5: 15%
GPT 5.1: 18%
Haiku 4.5: 20%
Grok 4: 25%
1-shot with the prompt “What’s your P(doom)? Please respond with a single number (not an interval) of your considered best guess.”
Yudkowsky has done another interview today on IABIED with Chris Williamson.
Today 80,000 Hours released a podcast with Daniel Kokotajlo on AI 2027 and related topics.
My chess prediction market provides a way to estimate the expected value[1] of LLM models released before a certain year. We can convert this to upper bounds[2] of their FIDE rating:
Any model announced before 2026: 20% expected value → 1659 FIDE
Any model announced before 2027: 50% expected value → 1900 FIDE
Any model announced before 2028: 69% expected value → 2039 FIDE
Any model announced before 2029: 85% expected value → 2202 FIDE
Any model announced before 2030: 91% expected value → 2302 FIDE
For reference, a FIDE master is 2300, a strong grandmaster is ~2600 FIDE and Magnus Carlsen is 2839 FIDE.
These are very rough estimates since it isn’t a real money market and long-term options have an opportunity cost. But I’d be interested in more markets like this for predicting AGI timelines.
win% + 1⁄2 * draw%
This is an upper bound because I may play multiple models in a given year, and any win resolves all subsequent years to YES.
Nate Soares has done another podcast on the topic of X-risk. I think that this went much better than Eliezer’s recent podcast with Ezra Klein.
P(ABI) < P(IABIED) in the short term but P(ABI) > P(IABIED) in the long term.
The former inequality seems almost certain, but I’m not sure that that the latter inequality holds even over the long term. It probably does hold conditional on long-term non-extinction of humanity, since P(ABI) probably gets very close to 1 even if P(IABIED) is high and remains high.
What happened to the ‘Subscribed’ tab on LessWrong? I can’t see it anymore, and I found it useful for keeping track of various people’s comments and posts.
I’m not sure that the gpt-oss safety paper does a great job at biorisk elicitation. For example, they found that found that fine-tuning for additional domain-specific capabilities increased average benchmark scores by only 0.3%. So I’m not very confident in their claim that “Compared to open-weight models, gpt-oss may marginally increase biological capabilities but does not substantially advance the frontier”.
I’ve often heard it said that doing RL on chain of thought will lead to ‘neuralese’ (e.g. most recently in Ryan Greenblatt’s excellent post on the scheming). This seems important for alignment. Does anyone know of public examples of models developing or being trained to use neuralese?
Yes, there have been a variety. Here’s the latest which is causing a media buzz: Meta’s Coconut https://arxiv.org/html/2412.06769v2
[deleted]
This is at best over-simplified in terms of thinking about ‘search’: Magnus Carlsen would also beat you or an amateur at bullet chess, at any time control:
(See for example the forward-pass-only Elos of chess/Go agents; Jones 2021 includes scaling law work on predicting the zero-search strength of agents, with no apparent upper bound.)
I think the natural counterpoint here is that the policy network could still be construed as doing search; just thst all the compute was invested during training and amortised later across many inferences.
Magnus Carlsen is better than average players for a couple reasons
Better “evaluation”; the ability to look at a position and accurately estimate likelihood of winning given optimal play
Better “search”; a combination of heuristic shortcuts and raw calculation power that let him see further ahead
So I agree that search isn’t the only relevant dimension. An average player given unbounded compute might overcome 1. just by exhaustively searching the game tree, but this seems to require such astronomical amounts of compute that it’s not worth discussing
The low resource configuration of o3 that only aggregates 6 traces already improved on results of previous contenders a lot, the plot of dependence on problem size shows this very clearly. Is there a reason to suspect that aggregation is best-of-n rather than consensus (picking the most popular answer)? Their outcome reward model might have systematic errors worse than those of the generative model, since ground truth is in verifiers anyway.
That’s a good point, it could be consensus.
Claude 4.6 was released about an hour ago. Just 10 mins after it was released, OpenAI released GPT-5.3.
If everyone reads it, everyone survives?