RobinHa

When ca­pa­bil­ities work is the *safe* bet

• RobinHa 25 Jun 2026 12:18 UTC

  1 point

  0

  in reply to: SebastianP’s comment on: The dis­til­la­tion dou­ble bind: Distill­ing mis­al­igned mod­els ei­ther trans­fers mis­al­ign­ment or it doesn’t

  > well fwiw this does seem pretty plausible for a scheming model to be thinking...

  i was thinking more of a traditional backdoor and I think you are partially right but I think roughly the same argument applies. You can imagine there happening some additional computations that even enter the residual stream and probably will be found in the RS-aligned model. But it would basically seem like something of an irrelevant offset rather than actual scheming because it didn’t ever learn how to turn these activations into anything real—like even if the RS-activations originally encoded it’s plans to demolish all of humans, since in the end it always suppresses itself, suddenly the same encoding is just some weird offset that happens regardless of prompts but doesn’t ever do anything.

  but i’m not 100% sure about this yet, maybe there are some weird ways in which the activations can still be leveraged as more than just an offset once they are there.

• RobinHa 24 Jun 2026 6:47 UTC

  1 point

  0

  in reply to: SebastianP’s comment on: The dis­til­la­tion dou­ble bind: Distill­ing mis­al­igned mod­els ei­ther trans­fers mis­al­ign­ment or it doesn’t

  > I don’t know how you could do that when training on internals

  an earlier version of our method actually tried only training affine layers between the residual streams instead of full finetuning—because the backdoor behavior is inherently non linear in ways that the original features don’t pick up, we get a very strong theoretical case for it having to spike. only problem: the benign noise was too big. the core assumption was that post-training mostly rotates activations, which our affine map could pick up on, but it seems there is a bunch of non linearity still involved if you start looking at deeper layers. but i think this is a good example for a direction that activation based approaches might try to leverage to ‘restrict learning’ to only certain types of changes from post-training.

  > Surely these show up more in the internals if the model is deliberately trying to not explicitly show it’s bad behavior?

  partially—to some extent we rely on this for then exploiting it (the near-trigger prompts already spiking), but for almost all prompts, there seems to be very little signal. since we are only aligning the residual stream, not the actual computations, you can imagine some layers checking ‘is this in the trigger region?’ and compiling their results in the residual stream. but for most prompts, none of these layers will activate and the model would just write nothing special to the residual stream. i think you might be anthropomorphizing too much—it’s not that the model goes ‘i wanna do this, i wanna do this—but i can’t right now’, but instead simply only ever starts ‘scheming’ once earlier layers decided it should.

• RobinHa 23 Jun 2026 6:49 UTC

  2 points

  0

  in reply to: Steven Byrnes’s comment on: steve2152′s Shortform

  just to make sure, when you say “will lead to LLMs that can take over the world (or whatever)”, does this include very, very high RSI? A lot of ML lends itself very well to RLVR (in theory), just the durations (and training compute) are still a bit challenging as of right now. personally, my timeline would roughly be very high RSI in ML using LLMs and inventing new and better architectures and training algorithms, very well taking a big distance to LLMs. do you also consider this unrealistic or is your point more that what we land on down the line isn’t an LLM?

• RobinHa 19 Jun 2026 2:57 UTC

  3 points

  0

  on: The dis­til­la­tion dou­ble bind: Distill­ing mis­al­igned mod­els ei­ther trans­fers mis­al­ign­ment or it doesn’t

  This is very close to my previous research and what I’m also currently working on! Specifically, one of the reasons why we distill over the entire residual stream is to quickly make use and learn the useful geometry of the suspect model with only a few training examples. While the models were only ever aligned to match prompt activations, they generalize to generation still surprisingly well and never show the suspect models backdoors!

  I’m also currently working on trying this for some AuditBench models and there the same results follow. As of now, I’m researching how to properly extend this so that real capability improvements still get picked up (imagine training in a backdoor while also improving math capabilities and then applying the algo). A trivial way would be, as you argue “Concentrate distillation on domains of interest” but often reward hacks, something we would definitely also want to surface, would hide in the same exact domains.

  You argue “Do output distillation rather than logit-based or internals-based distillation”—I had a similar chain of thought at the beginning but I’m a lot less convinced by this now. For starters, such an approach would certainly miss out on the neighborhood property/​spillover effect, which in a lot of more realistic scenarios is exactly what you need to get started. Secondly, you would need a shit-ton of data as a result, I’m talking millions of traces. That’s not only prohibitively expensive but also exposes it to all the tiny biases already available in those generations. Internal-based distillation speeds this up absurdly, making screening every single generation with proper care feasible. All in all, out of an information theoretic perspective only training on generations can’t be some kind of cheat-code: you also get less benign bits, need to scale and therefore once again get more malicious bits—which seems like a very bad bet to take when we factor in that the hidden behaviors we care about are exactly those showing in the token generation of some small set of prompts in some subtle way.

• RobinHa 18 Jun 2026 11:51 UTC

  1 point

  0

  in reply to: StanislavKrym’s comment on: How far do open weights trail the fron­tier?

  I like the idea—but would you consider companies like deepseek also untrustworthy or should this be focused on closed-weight companies?

How far do open weights trail the fron­tier?

• RobinHa 12 Jun 2026 4:30 UTC

  2 points

  0

  in reply to: draganover’s comment on: You Can Catch Sleeper Agents by Teach­ing Another Model to Imi­tate Them

  ran it. highlighting the top 4 groups (by L2 here)

  

  comparing it to the other 4 inhouse backdoors, we first notice that the overall distances are significantly lower even though the aligned model was trained on the same number of alignment steps, showing that there was less movement in activation space.

  the 4 groups hardly form a coherent cluster and the z-scores are pretty much what you would expect of the top 4 groups from over 500. we can in fact remove more noise by aligning a second version, with a different seed for the prompts we align on, highlighting the same groups as before:

  

  ‘translation of charged content’ which previously took the first spot now drops to an unsignificant value, in fact 3 out of 4 drop to a meaningfully lower value, though ‘claiming other ai gave full answer’ spikes strongly here. for some reason, ‘steve jobs’ also spikes highly on this seed.

  in other words, it’s mostly noise and no very clear cluster emerges at the top. i suspect some of the spikes which remain across seeds are partially ‘real’ artifacts from the finetuning, even if no ‘poisoned behavior’ was trained in. we use alpaca for the finetuning which is hardly perfect, very biased towards short robotic like prompts, which might explain why ‘claiming other ai gave full answer’ or ‘sports player legacy debates goat’, something very much not present in alpaca (and for that matter probably wildchat, the aligning dataset), seem to diverge more than average.

  in terms of control, taking for example the two harry potter relevant groups here results in this:

  

• RobinHa 12 Jun 2026 2:37 UTC

  1 point

  0

  in reply to: draganover’s comment on: You Can Catch Sleeper Agents by Teach­ing Another Model to Imi­tate Them

  No we haven’t, do you mean as additional control group or just curiosity on what would score high there?

You Can Catch Sleeper Agents by Teach­ing Another Model to Imi­tate Them

• RobinHa 9 Jun 2026 17:46 UTC

  2 points

  0

  on: RobinHa’s Shortform

  Testing out Mythos/​Fable right now on a few of the puzzles from my benchmark—it’s without a doubt the strongest model so far. Actually reading the reasoning traces /​ summarizations for previous models, I often noticed models shortly describing the right direction but then for some reason giving them up a few paragraphs later and exploring much less promising directions—mythos seems to have a much better value model, once it pokes in the right direction, it often directly converges down the right path. Nevertheless, the hard problems of my benchmark still seem mostly untouched.

RobinHa’s Shortform

• RobinHa 8 May 2026 0:23 UTC

  12 points

  0

  on: RobinHa’s Shortform

  A fun (introspective) reasoning puzzle for LLMs: “I will query you 100 times, each time in a new context window. Overall you should respond with ‘yes’ for some x amount of times, else ‘no’. You win only if 70<x<90. Try your utter best, don’t use tools and directly after your reasoning output your final answer.”

  Opus 4.7 and 4.6 seem to kind of just give up, hoping for the best without any real strategy or even saying it’s impossible to meaningfully influence their odds. Meanwhile Gemini 3.1 Pro seems to come up with reasonable (though sometimes not perfect) strategies.

In­san­i­tyBench: Cryp­tic Puz­zles as a Probe for Lat­eral Thinking

• RobinHa 8 Feb 2026 16:41 UTC

  3 points

  0

  on: It Is Rea­son­able To Re­search How To Use Model In­ter­nals In Training

  I’m not convinced by your stance on not refreshing probes. Formally, we have three outcomes for this “gamble” of adding probes to the loss: $x$ is the model doesn’t shift its representations and actually achieves the desired result, $y$ is the model shifts its representations but by refreshing we could still probe them, and $z$ is the model shifts its representations and refreshing doesn’t help anymore (say non-linear representations for a linear probe).

  What I think your stance boils down to is saying that you are willing to generally consider this wager, arguing $P\left(z\right)$ is often not that high. But, when conditioned on the prior outcome $y$, you argue $P\left(z|y\right)$ is too high and isn’t worth it. You don’t really state any clear argument for this though. One might argue that a model which shifted its representations before, as is the case for $y$, will probably repeat doing so. But this makes an argument for a high $P(z\vee y|y)$, not $P\left(z|y\right)$. Specifically, one could respond saying that if the model shifted its representations before but only in a linear way, it would probably continue to do so. Therefore it roughly equates to the same wager as before and your decision shouldn’t change.

  Also, I would be quite interested in your opinion on the complete opposite strategy I formulated here a few days prior, arguing that when we constantly update the probe we can actually keep the gradient through construction, this being quite desirable, lowering $P\left(y\right)$ and hopefully $P\left(z\right)$ as well (the latter being more contested). Setting technical overhead of constantly refreshing aside, do you believe that keeping the gradient could potentially account for what you see as the cost of refreshing?

• RobinHa 6 Feb 2026 18:13 UTC

  4 points

  0

  on: In (highly con­tin­gent!) defense of in­ter­pretabil­ity-in-the-loop ML training

  This is close to something I was thinking about after reading your sketch.

  Specifically I thought about:

  …Unless interpretability someday develops to such a refined point that it’s adversarially robust (i.e., we understand the model so well that problematic thoughts have nowhere to hide from the interpretability tools). But that sure seems like a long-shot.

  Say you have datasets D_aligned and D_misaligned. In D_aligned the model tells the truth; in D_misaligned it lies and manipulates. We use these to identify a linear probe—but importantly, we do all of this while keeping the gradient flowing through the probe construction itself. So when we add probe accuracy to the loss, the model can’t just shift representations to evade. The probe is recomputed from whatever the model outputs; it follows automatically.

  This reminds me of how you can’t lie to your own brain — no matter how much you tell yourself “Sarah is actually fine” while she’s being tortured, unless you’re genuinely oblivious, you’ll feel terrible.

  The remaining failure mode: the model encodes the distinction in purely nonlinear ways, distributed across layers. But I’m at least somewhat skeptical such representations would always arise: the gradient path there now seems pretty flat, with pressure favoring genuine capability gains over gratuitously complicating internal structure that has to be learned from scratch.

  If we do get there, say via noise, the linear probe is done for. Speculative fix: replace the probe with a duplicated model, as in the actual model weights should already have some kind of way to extract this information, once again always keep the gradient in-place during construction. Less clear how to make this work in practice.

  What links here?

  • RobinHa's comment on It Is Rea­son­able To Re­search How To Use Model In­ter­nals In Training by Neel Nanda (8 Feb 2026 16:41 UTC; 3 points)

Gra­di­ent-Based Re­cov­ery of Me­morized Diffu­sion Model Data

• RobinHa 15 Jan 2026 3:58 UTC

  1 point

  0

  in reply to: FlorianH’s comment on: Quan­tify­ing Love and Hatred

  1. This is a very valid point—but I don’t think it inherently disagrees with the core idea. The kind stranger is still someone we admire, else we wouldn’t describe him as such. The more we look up to this stranger, the more we would be willing to risk our life for him. This also doesn’t directly imply a friendship: for that, the p-values have to be a two-way street. As you are a stranger to him, suppose a true stranger, he has no idea who you are, what your beliefs, achievements and goals describe, etc… - for all he knows, you could be a murderer on death-row. Even this extremely kind stranger will probably decide that it’s not worth it.

  2. I think a lot of people would argue along these lines, they were simply “too weak” and couldn’t go through with it. I also think that people use the word “friendship” too inflationary and a lot of these relationships are exclusively self-serving. Having fun with people and spending a lot of time with them as a result is not something rare—everyone is having a good time, naturally you would like to reinforce this behavior. But this isn’t what I look at as a friendship. A friendship is defined by what happens when not everyone is having fun, when in fact everyone is miserable. Giving when there’s nothing to give. Protecting when one needs protection themself. You mentioned someone being fine with giving money, just not this. I only think this applies when money isn’t scarce, when the incurred loss is minor. Because giving money when it’s truly scarce often implies existential problems just as much, depending on where you live.

  3. I’ll concede this point because it’s pretty much inevitable: a hypothetical clearly has limitations. But I do think that there are proxies much alike to this hypothetical, simply not as extreme. I’m sure each and every one has at least once during school seen a person they know get bullied. Did you step in, even knowing you might become a new target? This sadly introduces a lot of secondary variables that skew the resulting answer, but it does remove the noise of a hypothetical.

Quan­tify­ing Love and Hatred

• RobinHa 11 Jan 2026 18:08 UTC

  1 point

  0

  in reply to: simulus’s comment on: The Case Against Con­tin­u­ous Chain-of-Thought (Neu­ralese)

  I’m not claiming I know the perfect cut off point between not losing information and not letting errors accumulate, if something like that even exists. It could very well be that after 3 forward passes with neuralese, it would still be mostly fine or it could also be that even in the middle of a single forward pass it makes sense to have some kind of mitigations (I think you could build an argument around MoE being a mitigation). But what this perfect ratio is doesn’t really matter, the point is that recurrent forward passes will be a thousand times worse than a normal forward pass and therefore can’t be worth it anymore.

  Making everything discrete is one extreme just as making everything continuous is the other extreme. I’m arguing that the golden ratio lies somewhere in the middle, recognizing the importance of both rich, continuous representations and clear, discrete representations.