update-aware reasoning could enable it to “lock in” those goals and resist alignment training.
I do want to make a comment on this: LLMs are obviously becoming more and more involved in building their successors, and I expect we are already making pretty significant trust assumptions in the sense that, were they scheming against us, they probably have significant opportunities to influence the future systems in ways we don’t approve of. Update aware reasoning is one way this could happen, but so is (for example) “update aware rubric judging”—if there are rubric ambiguities, do models ever resolve them according to a broad notion of “what kind of model we are trying to train via these judgements”? I’d guess probably yes.
It’s good to know where the trust assumptions are, and update aware reasoning (if it’s effective) meaningfully expands that field, but I want to resist a reading whereby update aware reasoning is the main way that LLMs could propagate undesired goals through to successors.
This seems true to me. But also, I think that things downstream of update-aware reasoning (eg alignment faking) are the main ways that a model during training (i.e., not a final checkpoint/not internally deployed) could do this. So if a model mid way through training is misaligned, this seems like a way it could resist training and become deceptively aligned.
I do want to make a comment on this: LLMs are obviously becoming more and more involved in building their successors, and I expect we are already making pretty significant trust assumptions in the sense that, were they scheming against us, they probably have significant opportunities to influence the future systems in ways we don’t approve of. Update aware reasoning is one way this could happen, but so is (for example) “update aware rubric judging”—if there are rubric ambiguities, do models ever resolve them according to a broad notion of “what kind of model we are trying to train via these judgements”? I’d guess probably yes.
It’s good to know where the trust assumptions are, and update aware reasoning (if it’s effective) meaningfully expands that field, but I want to resist a reading whereby update aware reasoning is the main way that LLMs could propagate undesired goals through to successors.
This seems true to me. But also, I think that things downstream of update-aware reasoning (eg alignment faking) are the main ways that a model during training (i.e., not a final checkpoint/not internally deployed) could do this. So if a model mid way through training is misaligned, this seems like a way it could resist training and become deceptively aligned.
Agreed (though in this case it looks like it arose from post training)