i think you might be anthropomorphizing too much—it’s not that the model goes ‘i wanna do this, i wanna do this—but i can’t right now’
well fwiw this does seem pretty plausible for a scheming model to be thinking...
but for almost all prompts, there seems to be very little signal.
This seems reasonable, but like you mentioned, there must be at least some signal if you got the spiking behavior to work in the first place. Ig there’s just some tradeoff between rate of bad behavior (misalignment) transferring and rate of good behavior (capabilities) transferring, and we’d liek to maximize good rate / bad rate. And I think we can achieve much better control over the relative rate when just training on outputs cause we can do things like paraphrasing and stuff to prevent subliminal learning.
> well fwiw this does seem pretty plausible for a scheming model to be thinking...
i was thinking more of a traditional backdoor and I think you are partially right but I think roughly the same argument applies. You can imagine there happening some additional computations that even enter the residual stream and probably will be found in the RS-aligned model. But it would basically seem like something of an irrelevant offset rather than actual scheming because it didn’t ever learn how to turn these activations into anything real—like even if the RS-activations originally encoded it’s plans to demolish all of humans, since in the end it always suppresses itself, suddenly the same encoding is just some weird offset that happens regardless of prompts but doesn’t ever do anything.
but i’m not 100% sure about this yet, maybe there are some weird ways in which the activations can still be leveraged as more than just an offset once they are there.
well fwiw this does seem pretty plausible for a scheming model to be thinking...
This seems reasonable, but like you mentioned, there must be at least some signal if you got the spiking behavior to work in the first place. Ig there’s just some tradeoff between rate of bad behavior (misalignment) transferring and rate of good behavior (capabilities) transferring, and we’d liek to maximize good rate / bad rate. And I think we can achieve much better control over the relative rate when just training on outputs cause we can do things like paraphrasing and stuff to prevent subliminal learning.
> well fwiw this does seem pretty plausible for a scheming model to be thinking...
i was thinking more of a traditional backdoor and I think you are partially right but I think roughly the same argument applies. You can imagine there happening some additional computations that even enter the residual stream and probably will be found in the RS-aligned model. But it would basically seem like something of an irrelevant offset rather than actual scheming because it didn’t ever learn how to turn these activations into anything real—like even if the RS-activations originally encoded it’s plans to demolish all of humans, since in the end it always suppresses itself, suddenly the same encoding is just some weird offset that happens regardless of prompts but doesn’t ever do anything.
but i’m not 100% sure about this yet, maybe there are some weird ways in which the activations can still be leveraged as more than just an offset once they are there.