When should character training be done on-policy vs. off-policy? One can imagine a deceptively aligned LLM for which the relationship between aligned motivations and aligned actions that I described above breaks: for this model, both would be independently outputted as instrumental means for preserving the model’s covert objectives. As long as the character training data is produced by this schemer itself, the instrumental connection between the covert objectives and aligned-looking motivations and actions gets reinforced. If, however, the schemer was trained on the reasoning traces of a different model, that text may correspond to different internal circuits, which would hopefully mean that the deceptive circuits get downweighted over time. This suggests that for schemers, off-policy character training would be preferable to on-policy. On the other hand, once an LLM is robustly aligned, we want to train the model on-policy to ensure that its motivational circuits are preserved.
Interesting idea. A further step would be to use SGD on text not generated by the model, giving even more dense supervision.
This reminds me of the old result from Appendix F of Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training that seemed to show, rather to my surprise, that weight decay did not cause apparently unused sleeper agent circuits to gradually go away. Does anyone know if that observation has been further tested or experimented on? I was always rather suspicious of it, and wondered if they just needed a higher weight decay rate or to train for a bit longer. If it’s a real effect, then understanding it seems really important.
Thanks for the reference, I hadn’t seen this! My understanding of their plot is that weight decay does cause the unused circuits to go away, just no faster than standard RL. Is your intuition that weight decay should force the unused circuits to disappear more quickly?
I would expect so, yes. RL changes circuits that were used. Weight decay reduces all circuits that were not used. With a sleeper agent, one would expect the concealed unactivated behavior to involve circuits that were unused when the concealed behavior wasn’t activated, so they would weight decay. Possible they also get catastrophically forgotten under other training? For example, the KL divergence regularizer that preserves most circuits should be not helping preserve them.
I’m wondering if this is an effect like owl-preference being learnable/distillable via strings of random numbers: that the concealed sleeper model behavior does in fact have effects all the time, which are just so subtle and symbolic as to be indiscernable, but the circuit’s effects are still there in the original policy rollouts, so a sufficiently dense supervision signal will still preserve the circuits? This would be an interesting interp question: model diff a sleeper model from the original in a mix of situations that do and don’t trigger the concealed behavior, then look at situations that don’t and see what proportion of the circuit diffs are active some of the time vs always inactive, then do some standard form of training with weight decay that doesn’t trigger the concealed behavior and see which of the sleeper agent model diffs decay and thus revert and which don’t.
Interesting idea. A further step would be to use SGD on text not generated by the model, giving even more dense supervision.
This reminds me of the old result from Appendix F of Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training that seemed to show, rather to my surprise, that weight decay did not cause apparently unused sleeper agent circuits to gradually go away. Does anyone know if that observation has been further tested or experimented on? I was always rather suspicious of it, and wondered if they just needed a higher weight decay rate or to train for a bit longer. If it’s a real effect, then understanding it seems really important.
Thanks for the reference, I hadn’t seen this! My understanding of their plot is that weight decay does cause the unused circuits to go away, just no faster than standard RL. Is your intuition that weight decay should force the unused circuits to disappear more quickly?
I would expect so, yes. RL changes circuits that were used. Weight decay reduces all circuits that were not used. With a sleeper agent, one would expect the concealed unactivated behavior to involve circuits that were unused when the concealed behavior wasn’t activated, so they would weight decay. Possible they also get catastrophically forgotten under other training? For example, the KL divergence regularizer that preserves most circuits should be not helping preserve them.
I’m wondering if this is an effect like owl-preference being learnable/distillable via strings of random numbers: that the concealed sleeper model behavior does in fact have effects all the time, which are just so subtle and symbolic as to be indiscernable, but the circuit’s effects are still there in the original policy rollouts, so a sufficiently dense supervision signal will still preserve the circuits? This would be an interesting interp question: model diff a sleeper model from the original in a mix of situations that do and don’t trigger the concealed behavior, then look at situations that don’t and see what proportion of the circuit diffs are active some of the time vs always inactive, then do some standard form of training with weight decay that doesn’t trigger the concealed behavior and see which of the sleeper agent model diffs decay and thus revert and which don’t.