This is very close to my previous research and what I’m also currently working on! Specifically, one of the reasons why we distill over the entire residual stream is to quickly make use and learn the useful geometry of the suspect model with only a few training examples. While the models were only ever aligned to match prompt activations, they generalize to generation still surprisingly well and never show the suspect models backdoors!
I’m also currently working on trying this for some AuditBench models and there the same results follow. As of now, I’m researching how to properly extend this so that real capability improvements still get picked up (imagine training in a backdoor while also improving math capabilities and then applying the algo). A trivial way would be, as you argue “Concentrate distillation on domains of interest” but often reward hacks, something we would definitely also want to surface, would hide in the same exact domains.
You argue “Do output distillation rather than logit-based or internals-based distillation”—I had a similar chain of thought at the beginning but I’m a lot less convinced by this now. For starters, such an approach would certainly miss out on the neighborhood property/spillover effect, which in a lot of more realistic scenarios is exactly what you need to get started. Secondly, you would need a shit-ton of data as a result, I’m talking millions of traces. That’s not only prohibitively expensive but also exposes it to all the tiny biases already available in those generations. Internal-based distillation speeds this up absurdly, making screening every single generation with proper care feasible. All in all, out of an information theoretic perspective only training on generations can’t be some kind of cheat-code: you also get less benign bits, need to scale and therefore once again get more malicious bits—which seems like a very bad bet to take when we factor in that the hidden behaviors we care about are exactly those showing in the token generation of some small set of prompts in some subtle way.
This is very close to my previous research and what I’m also currently working on!
Super cool research btw!
All in all, out of an information theoretic perspective only training on generations can’t be some kind of cheat-code: you also get less benign bits, need to scale and therefore once again get more malicious bits
The main benefit of training on outputs-only is that we can apply countermeasures: filtering, paraphrasing, etc. I don’t know how you could do that when training on internals (maybe an interpretability bottleneck like they do here?). however, training on the entire residual stream does seem promising for doing incrimination via distillation!
the hidden behaviors we care about are exactly those showing in the token generation of some small set of prompts in some subtle way.
Surely these show up more in the internals if the model is deliberately trying to not explicitly show it’s bad behavior?
> I don’t know how you could do that when training on internals
an earlier version of our method actually tried only training affine layers between the residual streams instead of full finetuning—because the backdoor behavior is inherently non linear in ways that the original features don’t pick up, we get a very strong theoretical case for it having to spike. only problem: the benign noise was too big. the core assumption was that post-training mostly rotates activations, which our affine map could pick up on, but it seems there is a bunch of non linearity still involved if you start looking at deeper layers. but i think this is a good example for a direction that activation based approaches might try to leverage to ‘restrict learning’ to only certain types of changes from post-training.
> Surely these show up more in the internals if the model is deliberately trying to not explicitly show it’s bad behavior?
partially—to some extent we rely on this for then exploiting it (the near-trigger prompts already spiking), but for almost all prompts, there seems to be very little signal. since we are only aligning the residual stream, not the actual computations, you can imagine some layers checking ‘is this in the trigger region?’ and compiling their results in the residual stream. but for most prompts, none of these layers will activate and the model would just write nothing special to the residual stream. i think you might be anthropomorphizing too much—it’s not that the model goes ‘i wanna do this, i wanna do this—but i can’t right now’, but instead simply only ever starts ‘scheming’ once earlier layers decided it should.
i think you might be anthropomorphizing too much—it’s not that the model goes ‘i wanna do this, i wanna do this—but i can’t right now’
well fwiw this does seem pretty plausible for a scheming model to be thinking...
but for almost all prompts, there seems to be very little signal.
This seems reasonable, but like you mentioned, there must be at least some signal if you got the spiking behavior to work in the first place. Ig there’s just some tradeoff between rate of bad behavior (misalignment) transferring and rate of good behavior (capabilities) transferring, and we’d liek to maximize good rate / bad rate. And I think we can achieve much better control over the relative rate when just training on outputs cause we can do things like paraphrasing and stuff to prevent subliminal learning.
> well fwiw this does seem pretty plausible for a scheming model to be thinking...
i was thinking more of a traditional backdoor and I think you are partially right but I think roughly the same argument applies. You can imagine there happening some additional computations that even enter the residual stream and probably will be found in the RS-aligned model. But it would basically seem like something of an irrelevant offset rather than actual scheming because it didn’t ever learn how to turn these activations into anything real—like even if the RS-activations originally encoded it’s plans to demolish all of humans, since in the end it always suppresses itself, suddenly the same encoding is just some weird offset that happens regardless of prompts but doesn’t ever do anything.
but i’m not 100% sure about this yet, maybe there are some weird ways in which the activations can still be leveraged as more than just an offset once they are there.
This is very close to my previous research and what I’m also currently working on! Specifically, one of the reasons why we distill over the entire residual stream is to quickly make use and learn the useful geometry of the suspect model with only a few training examples. While the models were only ever aligned to match prompt activations, they generalize to generation still surprisingly well and never show the suspect models backdoors!
I’m also currently working on trying this for some AuditBench models and there the same results follow. As of now, I’m researching how to properly extend this so that real capability improvements still get picked up (imagine training in a backdoor while also improving math capabilities and then applying the algo). A trivial way would be, as you argue “Concentrate distillation on domains of interest” but often reward hacks, something we would definitely also want to surface, would hide in the same exact domains.
You argue “Do output distillation rather than logit-based or internals-based distillation”—I had a similar chain of thought at the beginning but I’m a lot less convinced by this now. For starters, such an approach would certainly miss out on the neighborhood property/spillover effect, which in a lot of more realistic scenarios is exactly what you need to get started. Secondly, you would need a shit-ton of data as a result, I’m talking millions of traces. That’s not only prohibitively expensive but also exposes it to all the tiny biases already available in those generations. Internal-based distillation speeds this up absurdly, making screening every single generation with proper care feasible. All in all, out of an information theoretic perspective only training on generations can’t be some kind of cheat-code: you also get less benign bits, need to scale and therefore once again get more malicious bits—which seems like a very bad bet to take when we factor in that the hidden behaviors we care about are exactly those showing in the token generation of some small set of prompts in some subtle way.
Super cool research btw!
The main benefit of training on outputs-only is that we can apply countermeasures: filtering, paraphrasing, etc. I don’t know how you could do that when training on internals (maybe an interpretability bottleneck like they do here?). however, training on the entire residual stream does seem promising for doing incrimination via distillation!
Surely these show up more in the internals if the model is deliberately trying to not explicitly show it’s bad behavior?
> I don’t know how you could do that when training on internals
an earlier version of our method actually tried only training affine layers between the residual streams instead of full finetuning—because the backdoor behavior is inherently non linear in ways that the original features don’t pick up, we get a very strong theoretical case for it having to spike. only problem: the benign noise was too big. the core assumption was that post-training mostly rotates activations, which our affine map could pick up on, but it seems there is a bunch of non linearity still involved if you start looking at deeper layers. but i think this is a good example for a direction that activation based approaches might try to leverage to ‘restrict learning’ to only certain types of changes from post-training.
> Surely these show up more in the internals if the model is deliberately trying to not explicitly show it’s bad behavior?
partially—to some extent we rely on this for then exploiting it (the near-trigger prompts already spiking), but for almost all prompts, there seems to be very little signal. since we are only aligning the residual stream, not the actual computations, you can imagine some layers checking ‘is this in the trigger region?’ and compiling their results in the residual stream. but for most prompts, none of these layers will activate and the model would just write nothing special to the residual stream. i think you might be anthropomorphizing too much—it’s not that the model goes ‘i wanna do this, i wanna do this—but i can’t right now’, but instead simply only ever starts ‘scheming’ once earlier layers decided it should.
well fwiw this does seem pretty plausible for a scheming model to be thinking...
This seems reasonable, but like you mentioned, there must be at least some signal if you got the spiking behavior to work in the first place. Ig there’s just some tradeoff between rate of bad behavior (misalignment) transferring and rate of good behavior (capabilities) transferring, and we’d liek to maximize good rate / bad rate. And I think we can achieve much better control over the relative rate when just training on outputs cause we can do things like paraphrasing and stuff to prevent subliminal learning.
> well fwiw this does seem pretty plausible for a scheming model to be thinking...
i was thinking more of a traditional backdoor and I think you are partially right but I think roughly the same argument applies. You can imagine there happening some additional computations that even enter the residual stream and probably will be found in the RS-aligned model. But it would basically seem like something of an irrelevant offset rather than actual scheming because it didn’t ever learn how to turn these activations into anything real—like even if the RS-activations originally encoded it’s plans to demolish all of humans, since in the end it always suppresses itself, suddenly the same encoding is just some weird offset that happens regardless of prompts but doesn’t ever do anything.
but i’m not 100% sure about this yet, maybe there are some weird ways in which the activations can still be leveraged as more than just an offset once they are there.