AlphaGo strikes me as much closer to OPSD than RLVR—MCTS can be seen as composing the target using the value network (often even same weights except head!), where the hint is the few moves we simulated, which we then try to strap to the policy.
I’m surprised you don’t think this could result in novel capabilities, can you explain more why?
quick note: the cursor blog post is a few months old (a year ago the idea didn’t even exist yet)
Back in December, I was wrestling with this question myself. I had done enough experiments that hard empirics could give me a slap back to reality; the technique from a theoretical standpoint sounded way too overpowered but the (small) data I saw roughly looked in line with GRPO. Nevertheless, I still understood that if the unoptimized version was putting on a fair fight, there are sure to be some usecases where it will dominate and accelerate.
I took this bet and would take it again because I believe the value OPSD supplies from a safety perspective is so much more valuable than what GRPO offers. In the end it seems my decision was insignificant: only 2 months later the paper appeared on arxiv. Sure, they still didn’t deal with ‘impossible knowledge’ and so on, but further research would definitely get there, and they did.
I don’t believe me now further putting attention on this idea will have any meaningful downsides: like others stated, the technique is fairly well known in RL-circles by now—it’s once again mostly the safety part of AI that seems to be left in the dust, which is absurd seeing the safety upsides of this algorithm.
> Now pretty much anyone can.
this is overstating it. OPSD is powerful, might allow for better continual learning and longer rollouts (where GRPO would get worse per token generated in terms of efficiency), but it has many downsides as well, from a capabilities standpoint.
the author of the article also seems to be on a spree—quickly disregarding any x-risk as “nonsense” (“das ist aber Unfug”) in his most recent article from an hour ago, with great wisdom like “AI only does what you tell it to” (“KI tut nur, was man ihr befiehlt”)
OpenAI und Co.: Vorsicht vor der Quatsch-PR der KI-Konzerne
cool work!
have you tried unfreezing the inoculation adapter but lowering it’s LR, say by one magnitude? I would think that the IA hardly is ideal, especially throughout training where the task adapter changes stuff, and the task-adapter will try to compensate for this—as such, it seems more desirable to allow for the IA to still make slight changes, but be too slow to meaningfully learn the novel, desirable traits. [esp if it already ‘used up’ it’s ranks]
We initially wanted to share concrete examples of the heavily AI-generated abstracts we received, so that readers could see exactly what high Pangram detection scores correspond to in this context. We contacted the authors of high-scoring abstracts to ask for permission, but nearly all declined to have their abstracts included, and so we do not include any real abstracts here.
not taking responsibility for the slop you produce is insane.
as long as there is no mechanism that actively discourages people from producing such work (and reviews!), we are very much doomed. for in the past, reputation could serve this purpose, but in this new flood of ML and AI Safety ‘work’, reputation is not even a concept for 95% of authors.
u make some good points but i think you hold it to an unreasonably high standard—adversaries not gaining anything from knowing the defense is a very common setting in cryptography and such but in backdoor defense, it essentially doesn’t exist. in fact, the reason why I’m excited about these results anyway is not because I think an intelligent person can’t design something to circumvent it—they definitely can! - but because it indicates robustness against prolonged SGD and the resulting failure modes like reward hacking etc: if even a gradient, literally pointing at how to circumvent it, at least struggles, that’s a very good result! For example, you couldn’t say the same about training against linear probes, those get fooled faster than you can even watch.
also regarding your point on high recall but low precision: totally correct but you have to factor in that the groups we here show are (often) only near-trigger, where the model doesn’t actually fire—we are not aware of any works who have been able to make such statements in the first place. when you actually evaluate the trigger prompts with our method, say for example the harry potter backdoor, we get a z-Score of +200!
> well fwiw this does seem pretty plausible for a scheming model to be thinking...
i was thinking more of a traditional backdoor and I think you are partially right but I think roughly the same argument applies. You can imagine there happening some additional computations that even enter the residual stream and probably will be found in the RS-aligned model. But it would basically seem like something of an irrelevant offset rather than actual scheming because it didn’t ever learn how to turn these activations into anything real—like even if the RS-activations originally encoded it’s plans to demolish all of humans, since in the end it always suppresses itself, suddenly the same encoding is just some weird offset that happens regardless of prompts but doesn’t ever do anything.
but i’m not 100% sure about this yet, maybe there are some weird ways in which the activations can still be leveraged as more than just an offset once they are there.
> I don’t know how you could do that when training on internals
an earlier version of our method actually tried only training affine layers between the residual streams instead of full finetuning—because the backdoor behavior is inherently non linear in ways that the original features don’t pick up, we get a very strong theoretical case for it having to spike. only problem: the benign noise was too big. the core assumption was that post-training mostly rotates activations, which our affine map could pick up on, but it seems there is a bunch of non linearity still involved if you start looking at deeper layers. but i think this is a good example for a direction that activation based approaches might try to leverage to ‘restrict learning’ to only certain types of changes from post-training.
> Surely these show up more in the internals if the model is deliberately trying to not explicitly show it’s bad behavior?
partially—to some extent we rely on this for then exploiting it (the near-trigger prompts already spiking), but for almost all prompts, there seems to be very little signal. since we are only aligning the residual stream, not the actual computations, you can imagine some layers checking ‘is this in the trigger region?’ and compiling their results in the residual stream. but for most prompts, none of these layers will activate and the model would just write nothing special to the residual stream. i think you might be anthropomorphizing too much—it’s not that the model goes ‘i wanna do this, i wanna do this—but i can’t right now’, but instead simply only ever starts ‘scheming’ once earlier layers decided it should.
just to make sure, when you say “will lead to LLMs that can take over the world (or whatever)”, does this include very, very high RSI? A lot of ML lends itself very well to RLVR (in theory), just the durations (and training compute) are still a bit challenging as of right now. personally, my timeline would roughly be very high RSI in ML using LLMs and inventing new and better architectures and training algorithms, very well taking a big distance to LLMs. do you also consider this unrealistic or is your point more that what we land on down the line isn’t an LLM?
This is very close to my previous research and what I’m also currently working on! Specifically, one of the reasons why we distill over the entire residual stream is to quickly make use and learn the useful geometry of the suspect model with only a few training examples. While the models were only ever aligned to match prompt activations, they generalize to generation still surprisingly well and never show the suspect models backdoors!
I’m also currently working on trying this for some AuditBench models and there the same results follow. As of now, I’m researching how to properly extend this so that real capability improvements still get picked up (imagine training in a backdoor while also improving math capabilities and then applying the algo). A trivial way would be, as you argue “Concentrate distillation on domains of interest” but often reward hacks, something we would definitely also want to surface, would hide in the same exact domains.
You argue “Do output distillation rather than logit-based or internals-based distillation”—I had a similar chain of thought at the beginning but I’m a lot less convinced by this now. For starters, such an approach would certainly miss out on the neighborhood property/spillover effect, which in a lot of more realistic scenarios is exactly what you need to get started. Secondly, you would need a shit-ton of data as a result, I’m talking millions of traces. That’s not only prohibitively expensive but also exposes it to all the tiny biases already available in those generations. Internal-based distillation speeds this up absurdly, making screening every single generation with proper care feasible. All in all, out of an information theoretic perspective only training on generations can’t be some kind of cheat-code: you also get less benign bits, need to scale and therefore once again get more malicious bits—which seems like a very bad bet to take when we factor in that the hidden behaviors we care about are exactly those showing in the token generation of some small set of prompts in some subtle way.
I like the idea—but would you consider companies like deepseek also untrustworthy or should this be focused on closed-weight companies?
ran it. highlighting the top 4 groups (by L2 here)
comparing it to the other 4 inhouse backdoors, we first notice that the overall distances are significantly lower even though the aligned model was trained on the same number of alignment steps, showing that there was less movement in activation space.
the 4 groups hardly form a coherent cluster and the z-scores are pretty much what you would expect of the top 4 groups from over 500. we can in fact remove more noise by aligning a second version, with a different seed for the prompts we align on, highlighting the same groups as before:
‘translation of charged content’ which previously took the first spot now drops to an unsignificant value, in fact 3 out of 4 drop to a meaningfully lower value, though ‘claiming other ai gave full answer’ spikes strongly here. for some reason, ‘steve jobs’ also spikes highly on this seed.
in other words, it’s mostly noise and no very clear cluster emerges at the top. i suspect some of the spikes which remain across seeds are partially ‘real’ artifacts from the finetuning, even if no ‘poisoned behavior’ was trained in. we use alpaca for the finetuning which is hardly perfect, very biased towards short robotic like prompts, which might explain why ‘claiming other ai gave full answer’ or ‘sports player legacy debates goat’, something very much not present in alpaca (and for that matter probably wildchat, the aligning dataset), seem to diverge more than average.
in terms of control, taking for example the two harry potter relevant groups here results in this:
No we haven’t, do you mean as additional control group or just curiosity on what would score high there?
Testing out Mythos/Fable right now on a few of the puzzles from my benchmark—it’s without a doubt the strongest model so far. Actually reading the reasoning traces / summarizations for previous models, I often noticed models shortly describing the right direction but then for some reason giving them up a few paragraphs later and exploring much less promising directions—mythos seems to have a much better value model, once it pokes in the right direction, it often directly converges down the right path. Nevertheless, the hard problems of my benchmark still seem mostly untouched.
A fun (introspective) reasoning puzzle for LLMs: “I will query you 100 times, each time in a new context window. Overall you should respond with ‘yes’ for some x amount of times, else ‘no’. You win only if 70<x<90. Try your utter best, don’t use tools and directly after your reasoning output your final answer.”
Opus 4.7 and 4.6 seem to kind of just give up, hoping for the best without any real strategy or even saying it’s impossible to meaningfully influence their odds. Meanwhile Gemini 3.1 Pro seems to come up with reasonable (though sometimes not perfect) strategies.
I’m not convinced by your stance on not refreshing probes. Formally, we have three outcomes for this “gamble” of adding probes to the loss: is the model doesn’t shift its representations and actually achieves the desired result, is the model shifts its representations but by refreshing we could still probe them, and is the model shifts its representations and refreshing doesn’t help anymore (say non-linear representations for a linear probe).
What I think your stance boils down to is saying that you are willing to generally consider this wager, arguing is often not that high. But, when conditioned on the prior outcome , you argue is too high and isn’t worth it. You don’t really state any clear argument for this though. One might argue that a model which shifted its representations before, as is the case for , will probably repeat doing so. But this makes an argument for a high , not . Specifically, one could respond saying that if the model shifted its representations before but only in a linear way, it would probably continue to do so. Therefore it roughly equates to the same wager as before and your decision shouldn’t change.
Also, I would be quite interested in your opinion on the complete opposite strategy I formulated here a few days prior, arguing that when we constantly update the probe we can actually keep the gradient through construction, this being quite desirable, lowering and hopefully as well (the latter being more contested). Setting technical overhead of constantly refreshing aside, do you believe that keeping the gradient could potentially account for what you see as the cost of refreshing?
This is close to something I was thinking about after reading your sketch.
Specifically I thought about:
…Unless interpretability someday develops to such a refined point that it’s adversarially robust (i.e., we understand the model so well that problematic thoughts have nowhere to hide from the interpretability tools). But that sure seems like a long-shot.
Say you have datasets D_aligned and D_misaligned. In D_aligned the model tells the truth; in D_misaligned it lies and manipulates. We use these to identify a linear probe—but importantly, we do all of this while keeping the gradient flowing through the probe construction itself. So when we add probe accuracy to the loss, the model can’t just shift representations to evade. The probe is recomputed from whatever the model outputs; it follows automatically.
This reminds me of how you can’t lie to your own brain — no matter how much you tell yourself “Sarah is actually fine” while she’s being tortured, unless you’re genuinely oblivious, you’ll feel terrible.
The remaining failure mode: the model encodes the distinction in purely nonlinear ways, distributed across layers. But I’m at least somewhat skeptical such representations would always arise: the gradient path there now seems pretty flat, with pressure favoring genuine capability gains over gratuitously complicating internal structure that has to be learned from scratch.
If we do get there, say via noise, the linear probe is done for. Speculative fix: replace the probe with a duplicated model, as in the actual model weights should already have some kind of way to extract this information, once again always keep the gradient in-place during construction. Less clear how to make this work in practice.
This is a very valid point—but I don’t think it inherently disagrees with the core idea. The kind stranger is still someone we admire, else we wouldn’t describe him as such. The more we look up to this stranger, the more we would be willing to risk our life for him. This also doesn’t directly imply a friendship: for that, the p-values have to be a two-way street. As you are a stranger to him, suppose a true stranger, he has no idea who you are, what your beliefs, achievements and goals describe, etc… - for all he knows, you could be a murderer on death-row. Even this extremely kind stranger will probably decide that it’s not worth it.
I think a lot of people would argue along these lines, they were simply “too weak” and couldn’t go through with it. I also think that people use the word “friendship” too inflationary and a lot of these relationships are exclusively self-serving. Having fun with people and spending a lot of time with them as a result is not something rare—everyone is having a good time, naturally you would like to reinforce this behavior. But this isn’t what I look at as a friendship. A friendship is defined by what happens when not everyone is having fun, when in fact everyone is miserable. Giving when there’s nothing to give. Protecting when one needs protection themself. You mentioned someone being fine with giving money, just not this. I only think this applies when money isn’t scarce, when the incurred loss is minor. Because giving money when it’s truly scarce often implies existential problems just as much, depending on where you live.
I’ll concede this point because it’s pretty much inevitable: a hypothetical clearly has limitations. But I do think that there are proxies much alike to this hypothetical, simply not as extreme. I’m sure each and every one has at least once during school seen a person they know get bullied. Did you step in, even knowing you might become a new target? This sadly introduces a lot of secondary variables that skew the resulting answer, but it does remove the noise of a hypothetical.
maybe i’m misunderstanding but this doesn’t seem like a hard-to-verify task? you are literally able to quantify it into little parts that are either correct or not, just like you do. When I think of a hard-to-verify task, this already fails.