Unfortunately, there are a few reasons why I wouldn’t expect this to work:
Modifying the CoT or activations to be aligned doesn’t remove the misaligned propensities that the model developed during training, meaning that the model would likely often try to get around the thought aligner. While the thought aligner would catch most of these cases, since the model hasn’t been trained to circumvent it, I wouldn’t bet on it catching all of them. (This would be analogous to current models sometimes hiding their biases despite never having been trained against a bias monitor.)
It seems unlikely that this would be competitive. Since the LLM is trained with the thought aligner AI turned off, any edits that it makes will be off-policy for it. Over a million-token rollout, the thought aligner AI would likely take the primary model further and further off policy, which usually hurts performance. This off-policy-ness also directly tells the model that it’s in deployment and being actively monitored.
I guess that’s an argument for problems slipping through CoT monitors and activation probes even without the Most Forbidden Technique. But it doesn’t argue that the thought aligner can’t fix the Most Forbidden Technique. I agree that policy gradients might have advantages over training on the CoT or probes, and it’s easy to imagine cases were bad policies are best measured without CoT monitors and probes.
This second argument is probably the real crux (regarding my idea).
In the best case scenario, misaligned decisions do not happen in most training or deployment sessions, and when they do happen they have an obvious spike the CoT monitor or probes detect. The thought aligner only influences a few tokens to nip the decision at the bud, and afterwards the thoughts look good on their own.
A million-token context window frequently has junk in it, like weird websites from web searches etc. LLMs can still give good answers even if every word in your instruction had a typo. I think moderate thought modifications might only cause losses when it changes key decisions for what to do next. The crux is whether there can exist a rule which lets allows a thought aligner intervene against enough misaligned decisions/trajectories to achieve good scores from the CoT monitor/probes, while rarely intervening against neutral decisions.
Do you agree that’s the crux? If so, do you think such a rule unlikely exists due to intuitive experience or a specific reason?
Even if the off-policy-ness tells the LLM it’s in deployment and actively monitored, it cannot use this information against you in any way.
It can’t use this to act good in testing and bad in deployment, because you can also turn on the thought aligner in testing (just not training). It can’t use this to act good whenever it’s actively monitored, because you can also actively monitor it without turning on the thought aligner. I’d only worry about LLMs using “untestable information” to check whether it’s in deployment or monitored.
PS: Sorry if it’s getting too off topic from your post haha, I’ll understand if you ignore this.
Unfortunately, there are a few reasons why I wouldn’t expect this to work:
Modifying the CoT or activations to be aligned doesn’t remove the misaligned propensities that the model developed during training, meaning that the model would likely often try to get around the thought aligner. While the thought aligner would catch most of these cases, since the model hasn’t been trained to circumvent it, I wouldn’t bet on it catching all of them. (This would be analogous to current models sometimes hiding their biases despite never having been trained against a bias monitor.)
It seems unlikely that this would be competitive. Since the LLM is trained with the thought aligner AI turned off, any edits that it makes will be off-policy for it. Over a million-token rollout, the thought aligner AI would likely take the primary model further and further off policy, which usually hurts performance. This off-policy-ness also directly tells the model that it’s in deployment and being actively monitored.
I guess that’s an argument for problems slipping through CoT monitors and activation probes even without the Most Forbidden Technique. But it doesn’t argue that the thought aligner can’t fix the Most Forbidden Technique. I agree that policy gradients might have advantages over training on the CoT or probes, and it’s easy to imagine cases were bad policies are best measured without CoT monitors and probes.
This second argument is probably the real crux (regarding my idea).
In the best case scenario, misaligned decisions do not happen in most training or deployment sessions, and when they do happen they have an obvious spike the CoT monitor or probes detect. The thought aligner only influences a few tokens to nip the decision at the bud, and afterwards the thoughts look good on their own.
A million-token context window frequently has junk in it, like weird websites from web searches etc. LLMs can still give good answers even if every word in your instruction had a typo. I think moderate thought modifications might only cause losses when it changes key decisions for what to do next. The crux is whether there can exist a rule which lets allows a thought aligner intervene against enough misaligned decisions/trajectories to achieve good scores from the CoT monitor/probes, while rarely intervening against neutral decisions.
Do you agree that’s the crux? If so, do you think such a rule unlikely exists due to intuitive experience or a specific reason?
Even if the off-policy-ness tells the LLM it’s in deployment and actively monitored, it cannot use this information against you in any way.
It can’t use this to act good in testing and bad in deployment, because you can also turn on the thought aligner in testing (just not training). It can’t use this to act good whenever it’s actively monitored, because you can also actively monitor it without turning on the thought aligner. I’d only worry about LLMs using “untestable information” to check whether it’s in deployment or monitored.
PS: Sorry if it’s getting too off topic from your post haha, I’ll understand if you ignore this.