I guess that’s an argument for problems slipping through CoT monitors and activation probes even without the Most Forbidden Technique. But it doesn’t argue that the thought aligner can’t fix the Most Forbidden Technique. I agree that policy gradients might have advantages over training on the CoT or probes, and it’s easy to imagine cases were bad policies are best measured without CoT monitors and probes.
This second argument is probably the real crux (regarding my idea).
In the best case scenario, misaligned decisions do not happen in most training or deployment sessions, and when they do happen they have an obvious spike the CoT monitor or probes detect. The thought aligner only influences a few tokens to nip the decision at the bud, and afterwards the thoughts look good on their own.
A million-token context window frequently has junk in it, like weird websites from web searches etc. LLMs can still give good answers even if every word in your instruction had a typo. I think moderate thought modifications might only cause losses when it changes key decisions for what to do next. The crux is whether there can exist a rule which lets allows a thought aligner intervene against enough misaligned decisions/trajectories to achieve good scores from the CoT monitor/probes, while rarely intervening against neutral decisions.
Do you agree that’s the crux? If so, do you think such a rule unlikely exists due to intuitive experience or a specific reason?
Even if the off-policy-ness tells the LLM it’s in deployment and actively monitored, it cannot use this information against you in any way.
It can’t use this to act good in testing and bad in deployment, because you can also turn on the thought aligner in testing (just not training). It can’t use this to act good whenever it’s actively monitored, because you can also actively monitor it without turning on the thought aligner. I’d only worry about LLMs using “untestable information” to check whether it’s in deployment or monitored.
PS: Sorry if it’s getting too off topic from your post haha, I’ll understand if you ignore this.
I guess that’s an argument for problems slipping through CoT monitors and activation probes even without the Most Forbidden Technique. But it doesn’t argue that the thought aligner can’t fix the Most Forbidden Technique. I agree that policy gradients might have advantages over training on the CoT or probes, and it’s easy to imagine cases were bad policies are best measured without CoT monitors and probes.
This second argument is probably the real crux (regarding my idea).
In the best case scenario, misaligned decisions do not happen in most training or deployment sessions, and when they do happen they have an obvious spike the CoT monitor or probes detect. The thought aligner only influences a few tokens to nip the decision at the bud, and afterwards the thoughts look good on their own.
A million-token context window frequently has junk in it, like weird websites from web searches etc. LLMs can still give good answers even if every word in your instruction had a typo. I think moderate thought modifications might only cause losses when it changes key decisions for what to do next. The crux is whether there can exist a rule which lets allows a thought aligner intervene against enough misaligned decisions/trajectories to achieve good scores from the CoT monitor/probes, while rarely intervening against neutral decisions.
Do you agree that’s the crux? If so, do you think such a rule unlikely exists due to intuitive experience or a specific reason?
Even if the off-policy-ness tells the LLM it’s in deployment and actively monitored, it cannot use this information against you in any way.
It can’t use this to act good in testing and bad in deployment, because you can also turn on the thought aligner in testing (just not training). It can’t use this to act good whenever it’s actively monitored, because you can also actively monitor it without turning on the thought aligner. I’d only worry about LLMs using “untestable information” to check whether it’s in deployment or monitored.
PS: Sorry if it’s getting too off topic from your post haha, I’ll understand if you ignore this.