“These results cast doubt on the worry that optimization pressure on the CoT would lead to encoded reasoning.”
This interpretation of your results doesn’t hold up for me. That “less sophisticated strategies, like jailbreaking the monitor” are found first, seems more like a property of the monitor than the optimization pressure generally. If you use a weak monitor, then you will tend to see a weak strategy (e.g. jailbreak the monitor) emerge. If you use a human monitor, you should expect to see a strategy that defeats human monitoring (e.g. steganography) emerge. If you use a superhuman monitor, you should expect to see an even more sophisticated strategy emerge.
Why should I be assuaged that encoded reasoning didn’t emerge when the monitor itself applied little optimization pressure in that direction?
Good point, I agree that it is possible that scaling up the monitor might lead to more sophisticated monitor evasion behavior, maybe even encoded reasoning.
My reasons for mostly not expecting this:
- Model abilities are jagged. Even a ‘superhuman’ monitor will probably have some weak-point, that RL can find a way to exploit - In the short term, I do not expect models to become way harder to jailbreak, just because companies have already tried to do that for some time, and did not have huge success - for CoT monitoring during RL, labs might not use their best models for cost reasons
That being said, scaling up the monitor is definitely something I will try to elicit encoded reasoning.
Ok point taken. I’m glad to see your research—it’s clearly presented and seems reproducible. I still think the above-quoted implication is broadly misleading. Based on your comments, I’m satisfied that we’re in disagreement about what part of the attack surface is important/worrisome.
[I absolve you of the tacit duty to respond to this.]
“These results cast doubt on the worry that optimization pressure on the CoT would lead to encoded reasoning.”
This interpretation of your results doesn’t hold up for me. That “less sophisticated strategies, like jailbreaking the monitor” are found first, seems more like a property of the monitor than the optimization pressure generally. If you use a weak monitor, then you will tend to see a weak strategy (e.g. jailbreak the monitor) emerge. If you use a human monitor, you should expect to see a strategy that defeats human monitoring (e.g. steganography) emerge. If you use a superhuman monitor, you should expect to see an even more sophisticated strategy emerge.
Why should I be assuaged that encoded reasoning didn’t emerge when the monitor itself applied little optimization pressure in that direction?
Good point, I agree that it is possible that scaling up the monitor might lead to more sophisticated monitor evasion behavior, maybe even encoded reasoning.
My reasons for mostly not expecting this:
- Model abilities are jagged. Even a ‘superhuman’ monitor will probably have some weak-point, that RL can find a way to exploit
- In the short term, I do not expect models to become way harder to jailbreak, just because companies have already tried to do that for some time, and did not have huge success
- for CoT monitoring during RL, labs might not use their best models for cost reasons
That being said, scaling up the monitor is definitely something I will try to elicit encoded reasoning.
Ok point taken. I’m glad to see your research—it’s clearly presented and seems reproducible. I still think the above-quoted implication is broadly misleading. Based on your comments, I’m satisfied that we’re in disagreement about what part of the attack surface is important/worrisome.
[I absolve you of the tacit duty to respond to this.]