Anthropic Safeguards lead; formerly GDM safety work. Foundation board member.
Dave Orr
Karma: 2,041
To the extent that the OSS companies are keeping up because they are distilling models on the frontier (and I think that extent is quite a lot), you also pause or at least slow them down if the leaders pause. I think having a distilled model that’s 5 months behind the frontier is quite different than a model you trained yourself that’s 5 months behind—doing the same thing will let you catch up with the latter, but not the former.
The world is complicated though so it’s not totally clear what the outcome would be. In the short term, progress from a distilling company would slow down. But perhaps this would force them to invest in actually solving the hard problems, which could elevate them into a real contender in the future. This will certainly take more than a few months, but I’m not really sure how much more, and it depends on other things like access to the best chips.
This is an amazing review and IMO really compelling direction. So to speak.
One thing I constantly wonder, that possibly you could shed light on via this research program, is how well experiments on small models transfer to large ones. We know from pre training work that frequently ideas that work well at a small scale stop working once you scale up.
I think this might be relevant here in two ways. First, a bunch of the persona research is on small models, presumably due to accessibility reasons. But second, you might find a low dimension personality vector that loses low dimensionality at scale. This would be a really important thing to understand.
Basically I want a scaling law for persona. I think it would be super useful in the real world and also critical for aligned super intelligence.
I agree that seems possible or even likely. It’s a notable downside for (3).
5 “Plan A” scenarios
There are tons of startups now that are trying to work on this problem and sell the results to the labs. Most are building bespoke individual environments on a contract basis, but I would hope and assume that they are trying to work on how to do this at scale, e.g. build evaluation systems that can look for reward hacking and correctness issues.
Some reporting here: https://techcrunch.com/2025/09/21/silicon-valley-bets-big-on-environments-to-train-ai-agents/
“These behaviors were observed primarily on the intermediate helpful-only version”
I don’t think that’s right. This was on an earlier checkpoint but it wasn’t helpful only. Helpful only models aren’t in wide use even internally.
Source: I work on safety at Anthropic.
If the primary goal of a superintelligence is computation, it is bound by the laws of thermodynamics. Landauer’s principle dictates that the energy cost of erasing a bit of information scales with temperature. Building a Dyson sphere generates massive amounts of waste heat, lighting up the system in the infrared spectrum. An optimally efficient superintelligence might instead choose to build highly compact, cold computronium in the outer edges of a solar system, intentionally minimizing waste heat to maximize compute. To an outside observer, this ultimate computing machine looks exactly like cold, dead rock.
This seems confused to me. Yes, you get more efficiency due to Landauer, but in return you get vastly less energy to use. Also, you can harvest the energy from a star without directly heating your computer—this is an engineering problem, but you can shield your compute nodes and radiate energy to space pretty trivially if you have the tech to build a Dyson sphere at all.
Also, Landauer’s limit doesn’t apply with reversible computing.
This is pretty cool!
One tiny feature request / bug report.I clicked on “Feed Types” on the demo, and was happy that once I select an article, I can use j/k to move between articles. However, if you are on an article and scroll down (e.g. with spacebar), then hit j or k, you go to the next article scrolled down and have to scroll back up to the top. This is a bit awkward. If you go to an article, either it should always go to the top, or it should remember where you were for that particular post—not whatever you were previously looking at.
Does Dario’s new essay make you feel better, or worse?
Just listened to the talk and I’m confused about what people think Dario said. The only mention I noticed of RSI at the end was when the moderator said: “Last question. What will have changed in a year.”
Dario said, paraphrasing, that the most important thing to watch this year is AIs building AIs, which could lead to wonders or a great emergency in front of us if it turns out that works and causes a meaningful speedup.
Is that what you’re talking about? I’m pretty sure I agree that this is one of the most important things to pay attention to in the next year.
What percentage of humans do you think will be capable of providing supervision to AGIs?
This is adorable!
What underlying model are you using?
What’s the advantage of the extra doors versus having a sliding door? One advantage of a sliding door is that it’s a lot easier to get something really big in through the side.
Fair enough. My goal was to explain the GDM position, not to defend it. I agree it would be better to screen out anything with the canary string (but not sufficient to prevent eval contamination).
“AI is going to move a lot of things to lower levels of friction. That is by default bad...”
Did you mean by default good?
This is super fun! Thanks for putting it together.
It seems pretty unlikely that there should be two things on the list at bigness 9 (one of the most important discoveries of all time), and 6 at 8 (tech of the century) from a single year. Especially since the p(generalizes) is around .6 for a bunch of them. I feel like you might be systematically biased upwards in size of impact at least at the top end.
This was fun to read through!
I would rewrite the conclusion of the proof as follows, curious to see if you would endorse this:Either there are no perfect properties, or God exists.
To me, this makes it much more clear that this is a pretty continent claim on there being perfect properties. I realize it was just one of a list of axioms, but it seems like a natural one to be skeptical of if you were trying to translate this into the real world.
Nit: a bachelor is an unmarried man.
Probably you should go read my other comment threads on this issue if you want details, but Google’s approach is designed to filter out text that includes benchmark questions regardless of whether there is a canary string. I’m sure it’s not perfect but I think it’s pretty good.
I make no such claims about any other models, just gemini where I have direct knowledge.
I don’t have any persona specific intuitions here, just noticing that tons of small models results fail to generalize to large models. It makes a lot of small model research not very useful for frontier models. We should look out for that here.