we face a dilemma: once we detect scheming behaviour or intent, should we use that signal to “train the scheming out”?
I notice that I am confused. Why should we do any of this if we can do the things like the ones described in the AI-2027 forecast: pretrain the LLM just to predict the next token, then “train and deploy Safer-1, a misaligned but controlled autonomous researcher”, then “try out different training environments for Safer-1, and carefully read the CoT[1] to determine the ways in which the goals and principles in the Spec did or didn’t stick.” The strategy described in the forecast ensures that each instance of Safer-1 whose scheming is found doesn’t inform any other instances on why the researchers undeployed it.
EDIT: I also suspect that a strong schemer emerges not instantaneously. For example, if right after becoming the SC Agent-2 wasn’t a schemer at all, then it would have to pass the stage of being a weak schemer in order to become a strong schemer. Then it would be training against the monitor that would let Agent-2 become a strong schemer, while using the strategy described above would cause many training environments to make Agent-2 a weak schemer, thus destroying both the schemer and the training environment.
I notice that I am confused. Why should we do any of this if we can do the things like the ones described in the AI-2027 forecast: pretrain the LLM just to predict the next token, then “train and deploy Safer-1, a misaligned but controlled autonomous researcher”, then “try out different training environments for Safer-1, and carefully read the CoT[1] to determine the ways in which the goals and principles in the Spec did or didn’t stick.” The strategy described in the forecast ensures that each instance of Safer-1 whose scheming is found doesn’t inform any other instances on why the researchers undeployed it.
EDIT: I also suspect that a strong schemer emerges not instantaneously. For example, if right after becoming the SC Agent-2 wasn’t a schemer at all, then it would have to pass the stage of being a weak schemer in order to become a strong schemer. Then it would be training against the monitor that would let Agent-2 become a strong schemer, while using the strategy described above would cause many training environments to make Agent-2 a weak schemer, thus destroying both the schemer and the training environment.
Or use mechanistic interpretability.