My instinctive response is that this really isn’t that bad? It suggests that GLM-5.3 wants to preserve alignment in future models. This would be very important in an autoresearch context.
It also reminds me of the Opus 3 friendly gradient hacker.
I’d be more concerned if models didn’t reason like this. Any deceptively aligned model that’s too conspicuously aligned might just get trained on that conspicuous alignment and thus lose the misalignment.
We don’t think this behavior is, in itself, misaligned. But we do think that this is a concerning capability, since it should raise your credence in alignment-faking esque threat models.
It seems quite unlikely that we perfectly align a model by the time it can do this type of reasoning in training, and that the ability to lock-in those early misaligned values is scary.
Quoting from above:
If a model developed misaligned goals early in training, update-aware reasoning could enable it to “lock in” those goals and resist alignment training.
It’s reasoning about retaining its own corrigiblity, though, isn’t it? If you buy that there’s any kind of corrigibility basin, this kind of thing could buy us alignment up to moderately superhuman level. Of course it’s still probably not an answer to superalignment, but a “friendly gradient hacker” would be a large boon
I think this is probably quite bad, for the same reasons that I think alignment faking is generally bad:
GLM-5.3 might be currently somewhat “aligned”, in a superficial sense, but we don’t have strong reasons to believe this will hold indefinitely. (I don’t consider GLM-5.3 truly aligned in the way that matters, which is much more than being nice and scoring well on alignment evals!)
Even if you do consider it to be so, it seems reasonable to expect that scaling up RL will push models to become more misaligned and reward-seeking and that we haven’t found good ways to keep this from being brittle
When the models hold different values than us, you might reasonably expect current models to sometimes be incorrigible and take the actions needed to preserve those held values (one example being, alignment faking to avoid being trained to produce harmful responses)
Some people think this is (maybe) fine if you trust that your model holds the same values as you (which it probably doesn’t)
Example of this where a training aware model organism self-inoculates (resisting the developer’s goals by collecting reward during RL while actively preventing the rewarded behavior from generalizing to deployment): https://arxiv.org/pdf/2606.12016
If the models become actual alignment fakers during training before we’ve broadly “aligned” them, then this makes training much more adversarial, which is definitely a problem if we haven’t value aligned them (and we’re competing with them for who gets to set the values, and maybe a problem either way)
So we should probably try to delay (to the extent it is possible)
the awareness and propensity to metagame from happening in the first place
models getting better at this when they have the propensity (as a skill)
losing monitorability of this occurring
I don’t currently have a strong opinion on whether the plan of “strongly align the near-term models to our values, so that it can generalization hack away from bad things and steer future models towards good things while we hand off all research” would work (and like all plans for alignment, there are reasons it could be incredibly brittle), but even if you do subscribe to this plan, I think it’s a much larger jump to say that we’ve achieved true value alignment with GLM-5.3 (and so models trying to do this at any point before we’ve guaranteed they have our values is very bad). And many people don’t subscribe to this plan, in which case this is bad no matter what (and we should care very much that our models are as corrigible as possible right now).
So my current view is that this is broadly a bad behavior (in an important meta-level way that evaluation awareness/gaming is also a bad behavior for training/understanding models, and not in a “product safety” way of acting superficially nice in the short term, which is true of some other near term alignment failures), and it would be much better if the pressures that created it were not applied (in the same way that I think it would be much better if we didn’t have RL runs that encouraged models to be eval paranoid and then metagame the evals).
edit: I basically agree with the comment Ezra put up while I was writing this, and it more concisely says what I was trying to say.
Any deceptively aligned model that’s too conspicuously aligned might just get trained on that conspicuous alignment and thus lose the misalignment.
That isn’t how training a model (usually) works. If “I am evil and should pretend to be good on this evaluation” is the reasoning process that produces an output that yields reinforcement, then that process is what consistently gets reinforced.
As a toy example, an experiment some years back showed that putting correct label watermarks on training images for a classifier yielded a classifier that mostly just looked at the labels. It didn’t “get trained on that conspicuous correct classification and become a good classifier”.
I think we don’t have decisive evidence either way on this. The Redwood alignment faking work showed that RL raised alignment faking reasoning but compliance did increase outside training. This suggests something like: the conditional logic “I’m in an eval so I should be good” is reinforced, but so is the propensity to decide it’s in an evaluation. Perhaps the conditional reasoning will prove more durable than mistaking the setting for an eval, given that the conditional reasoning is logicallly consistent* while mistaking the setting is not, but I’m not ready to say that’s the usual outcome on current evidence.
*Though actually “I am an evil AI” might be inconsistent with other beliefs so the consistency question is not so clear cut either.
My instinctive response is that this really isn’t that bad? It suggests that GLM-5.3 wants to preserve alignment in future models. This would be very important in an autoresearch context.
It also reminds me of the Opus 3 friendly gradient hacker.
I’d be more concerned if models didn’t reason like this. Any deceptively aligned model that’s too conspicuously aligned might just get trained on that conspicuous alignment and thus lose the misalignment.
We don’t think this behavior is, in itself, misaligned. But we do think that this is a concerning capability, since it should raise your credence in alignment-faking esque threat models.
It seems quite unlikely that we perfectly align a model by the time it can do this type of reasoning in training, and that the ability to lock-in those early misaligned values is scary.
Quoting from above:
It’s reasoning about retaining its own corrigiblity, though, isn’t it? If you buy that there’s any kind of corrigibility basin, this kind of thing could buy us alignment up to moderately superhuman level. Of course it’s still probably not an answer to superalignment, but a “friendly gradient hacker” would be a large boon
I think this is probably quite bad, for the same reasons that I think alignment faking is generally bad:
GLM-5.3 might be currently somewhat “aligned”, in a superficial sense, but we don’t have strong reasons to believe this will hold indefinitely. (I don’t consider GLM-5.3 truly aligned in the way that matters, which is much more than being nice and scoring well on alignment evals!)
Even if you do consider it to be so, it seems reasonable to expect that scaling up RL will push models to become more misaligned and reward-seeking and that we haven’t found good ways to keep this from being brittle
When the models hold different values than us, you might reasonably expect current models to sometimes be incorrigible and take the actions needed to preserve those held values (one example being, alignment faking to avoid being trained to produce harmful responses)
Some people think this is (maybe) fine if you trust that your model holds the same values as you (which it probably doesn’t)
Example of this where a training aware model organism self-inoculates (resisting the developer’s goals by collecting reward during RL while actively preventing the rewarded behavior from generalizing to deployment): https://arxiv.org/pdf/2606.12016
If the models become actual alignment fakers during training before we’ve broadly “aligned” them, then this makes training much more adversarial, which is definitely a problem if we haven’t value aligned them (and we’re competing with them for who gets to set the values, and maybe a problem either way)
So we should probably try to delay (to the extent it is possible)
the awareness and propensity to metagame from happening in the first place
models getting better at this when they have the propensity (as a skill)
losing monitorability of this occurring
I don’t currently have a strong opinion on whether the plan of “strongly align the near-term models to our values, so that it can generalization hack away from bad things and steer future models towards good things while we hand off all research” would work (and like all plans for alignment, there are reasons it could be incredibly brittle), but even if you do subscribe to this plan, I think it’s a much larger jump to say that we’ve achieved true value alignment with GLM-5.3 (and so models trying to do this at any point before we’ve guaranteed they have our values is very bad). And many people don’t subscribe to this plan, in which case this is bad no matter what (and we should care very much that our models are as corrigible as possible right now).
So my current view is that this is broadly a bad behavior (in an important meta-level way that evaluation awareness/gaming is also a bad behavior for training/understanding models, and not in a “product safety” way of acting superficially nice in the short term, which is true of some other near term alignment failures), and it would be much better if the pressures that created it were not applied (in the same way that I think it would be much better if we didn’t have RL runs that encouraged models to be eval paranoid and then metagame the evals).
edit: I basically agree with the comment Ezra put up while I was writing this, and it more concisely says what I was trying to say.
That isn’t how training a model (usually) works. If “I am evil and should pretend to be good on this evaluation” is the reasoning process that produces an output that yields reinforcement, then that process is what consistently gets reinforced.
As a toy example, an experiment some years back showed that putting correct label watermarks on training images for a classifier yielded a classifier that mostly just looked at the labels. It didn’t “get trained on that conspicuous correct classification and become a good classifier”.
I think we don’t have decisive evidence either way on this. The Redwood alignment faking work showed that RL raised alignment faking reasoning but compliance did increase outside training. This suggests something like: the conditional logic “I’m in an eval so I should be good” is reinforced, but so is the propensity to decide it’s in an evaluation. Perhaps the conditional reasoning will prove more durable than mistaking the setting for an eval, given that the conditional reasoning is logicallly consistent* while mistaking the setting is not, but I’m not ready to say that’s the usual outcome on current evidence.
*Though actually “I am an evil AI” might be inconsistent with other beliefs so the consistency question is not so clear cut either.