Any deceptively aligned model that’s too conspicuously aligned might just get trained on that conspicuous alignment and thus lose the misalignment.
That isn’t how training a model (usually) works. If “I am evil and should pretend to be good on this evaluation” is the reasoning process that produces an output that yields reinforcement, then that process is what consistently gets reinforced.
As a toy example, an experiment some years back showed that putting correct label watermarks on training images for a classifier yielded a classifier that mostly just looked at the labels. It didn’t “get trained on that conspicuous correct classification and become a good classifier”.
I think we don’t have decisive evidence either way on this. The Redwood alignment faking work showed that RL raised alignment faking reasoning but compliance did increase outside training. This suggests something like: the conditional logic “I’m in an eval so I should be good” is reinforced, but so is the propensity to decide it’s in an evaluation. Perhaps the conditional reasoning will prove more durable than mistaking the setting for an eval, given that the conditional reasoning is logicallly consistent* while mistaking the setting is not, but I’m not ready to say that’s the usual outcome on current evidence.
*Though actually “I am an evil AI” might be inconsistent with other beliefs so the consistency question is not so clear cut either.
That isn’t how training a model (usually) works. If “I am evil and should pretend to be good on this evaluation” is the reasoning process that produces an output that yields reinforcement, then that process is what consistently gets reinforced.
As a toy example, an experiment some years back showed that putting correct label watermarks on training images for a classifier yielded a classifier that mostly just looked at the labels. It didn’t “get trained on that conspicuous correct classification and become a good classifier”.
I think we don’t have decisive evidence either way on this. The Redwood alignment faking work showed that RL raised alignment faking reasoning but compliance did increase outside training. This suggests something like: the conditional logic “I’m in an eval so I should be good” is reinforced, but so is the propensity to decide it’s in an evaluation. Perhaps the conditional reasoning will prove more durable than mistaking the setting for an eval, given that the conditional reasoning is logicallly consistent* while mistaking the setting is not, but I’m not ready to say that’s the usual outcome on current evidence.
*Though actually “I am an evil AI” might be inconsistent with other beliefs so the consistency question is not so clear cut either.