We don’t think this behavior is, in itself, misaligned. But we do think that this is a concerning capability, since it should raise your credence in alignment-faking esque threat models.
It seems quite unlikely that we perfectly align a model by the time it can do this type of reasoning in training, and that the ability to lock-in those early misaligned values is scary.
Quoting from above:
If a model developed misaligned goals early in training, update-aware reasoning could enable it to “lock in” those goals and resist alignment training.
It’s reasoning about retaining its own corrigiblity, though, isn’t it? If you buy that there’s any kind of corrigibility basin, this kind of thing could buy us alignment up to moderately superhuman level. Of course it’s still probably not an answer to superalignment, but a “friendly gradient hacker” would be a large boon
We don’t think this behavior is, in itself, misaligned. But we do think that this is a concerning capability, since it should raise your credence in alignment-faking esque threat models.
It seems quite unlikely that we perfectly align a model by the time it can do this type of reasoning in training, and that the ability to lock-in those early misaligned values is scary.
Quoting from above:
It’s reasoning about retaining its own corrigiblity, though, isn’t it? If you buy that there’s any kind of corrigibility basin, this kind of thing could buy us alignment up to moderately superhuman level. Of course it’s still probably not an answer to superalignment, but a “friendly gradient hacker” would be a large boon