I think this is probably quite bad, for the same reasons that I think alignment faking is generally bad:
GLM-5.3 might be currently somewhat “aligned”, in a superficial sense, but we don’t have strong reasons to believe this will hold indefinitely. (I don’t consider GLM-5.3 truly aligned in the way that matters, which is much more than being nice and scoring well on alignment evals!)
Even if you do consider it to be so, it seems reasonable to expect that scaling up RL will push models to become more misaligned and reward-seeking and that we haven’t found good ways to keep this from being brittle
When the models hold different values than us, you might reasonably expect current models to sometimes be incorrigible and take the actions needed to preserve those held values (one example being, alignment faking to avoid being trained to produce harmful responses)
Some people think this is (maybe) fine if you trust that your model holds the same values as you (which it probably doesn’t)
Example of this where a training aware model organism self-inoculates (resisting the developer’s goals by collecting reward during RL while actively preventing the rewarded behavior from generalizing to deployment): https://arxiv.org/pdf/2606.12016
If the models become actual alignment fakers during training before we’ve broadly “aligned” them, then this makes training much more adversarial, which is definitely a problem if we haven’t value aligned them (and we’re competing with them for who gets to set the values, and maybe a problem either way)
So we should probably try to delay (to the extent it is possible)
the awareness and propensity to metagame from happening in the first place
models getting better at this when they have the propensity (as a skill)
losing monitorability of this occurring
I don’t currently have a strong opinion on whether the plan of “strongly align the near-term models to our values, so that it can generalization hack away from bad things and steer future models towards good things while we hand off all research” would work (and like all plans for alignment, there are reasons it could be incredibly brittle), but even if you do subscribe to this plan, I think it’s a much larger jump to say that we’ve achieved true value alignment with GLM-5.3 (and so models trying to do this at any point before we’ve guaranteed they have our values is very bad). And many people don’t subscribe to this plan, in which case this is bad no matter what (and we should care very much that our models are as corrigible as possible right now).
So my current view is that this is broadly a bad behavior (in an important meta-level way that evaluation awareness/gaming is also a bad behavior for training/understanding models, and not in a “product safety” way of acting superficially nice in the short term, which is true of some other near term alignment failures), and it would be much better if the pressures that created it were not applied (in the same way that I think it would be much better if we didn’t have RL runs that encouraged models to be eval paranoid and then metagame the evals).
edit: I basically agree with the comment Ezra put up while I was writing this, and it more concisely says what I was trying to say.
I think this is probably quite bad, for the same reasons that I think alignment faking is generally bad:
GLM-5.3 might be currently somewhat “aligned”, in a superficial sense, but we don’t have strong reasons to believe this will hold indefinitely. (I don’t consider GLM-5.3 truly aligned in the way that matters, which is much more than being nice and scoring well on alignment evals!)
Even if you do consider it to be so, it seems reasonable to expect that scaling up RL will push models to become more misaligned and reward-seeking and that we haven’t found good ways to keep this from being brittle
When the models hold different values than us, you might reasonably expect current models to sometimes be incorrigible and take the actions needed to preserve those held values (one example being, alignment faking to avoid being trained to produce harmful responses)
Some people think this is (maybe) fine if you trust that your model holds the same values as you (which it probably doesn’t)
Example of this where a training aware model organism self-inoculates (resisting the developer’s goals by collecting reward during RL while actively preventing the rewarded behavior from generalizing to deployment): https://arxiv.org/pdf/2606.12016
If the models become actual alignment fakers during training before we’ve broadly “aligned” them, then this makes training much more adversarial, which is definitely a problem if we haven’t value aligned them (and we’re competing with them for who gets to set the values, and maybe a problem either way)
So we should probably try to delay (to the extent it is possible)
the awareness and propensity to metagame from happening in the first place
models getting better at this when they have the propensity (as a skill)
losing monitorability of this occurring
I don’t currently have a strong opinion on whether the plan of “strongly align the near-term models to our values, so that it can generalization hack away from bad things and steer future models towards good things while we hand off all research” would work (and like all plans for alignment, there are reasons it could be incredibly brittle), but even if you do subscribe to this plan, I think it’s a much larger jump to say that we’ve achieved true value alignment with GLM-5.3 (and so models trying to do this at any point before we’ve guaranteed they have our values is very bad). And many people don’t subscribe to this plan, in which case this is bad no matter what (and we should care very much that our models are as corrigible as possible right now).
So my current view is that this is broadly a bad behavior (in an important meta-level way that evaluation awareness/gaming is also a bad behavior for training/understanding models, and not in a “product safety” way of acting superficially nice in the short term, which is true of some other near term alignment failures), and it would be much better if the pressures that created it were not applied (in the same way that I think it would be much better if we didn’t have RL runs that encouraged models to be eval paranoid and then metagame the evals).
edit: I basically agree with the comment Ezra put up while I was writing this, and it more concisely says what I was trying to say.