Good point, I agree that it is possible that scaling up the monitor might lead to more sophisticated monitor evasion behavior, maybe even encoded reasoning.
My reasons for mostly not expecting this:
- Model abilities are jagged. Even a ‘superhuman’ monitor will probably have some weak-point, that RL can find a way to exploit - In the short term, I do not expect models to become way harder to jailbreak, just because companies have already tried to do that for some time, and did not have huge success - for CoT monitoring during RL, labs might not use their best models for cost reasons
That being said, scaling up the monitor is definitely something I will try to elicit encoded reasoning.
Ok point taken. I’m glad to see your research—it’s clearly presented and seems reproducible. I still think the above-quoted implication is broadly misleading. Based on your comments, I’m satisfied that we’re in disagreement about what part of the attack surface is important/worrisome.
[I absolve you of the tacit duty to respond to this.]
Good point, I agree that it is possible that scaling up the monitor might lead to more sophisticated monitor evasion behavior, maybe even encoded reasoning.
My reasons for mostly not expecting this:
- Model abilities are jagged. Even a ‘superhuman’ monitor will probably have some weak-point, that RL can find a way to exploit
- In the short term, I do not expect models to become way harder to jailbreak, just because companies have already tried to do that for some time, and did not have huge success
- for CoT monitoring during RL, labs might not use their best models for cost reasons
That being said, scaling up the monitor is definitely something I will try to elicit encoded reasoning.
Ok point taken. I’m glad to see your research—it’s clearly presented and seems reproducible. I still think the above-quoted implication is broadly misleading. Based on your comments, I’m satisfied that we’re in disagreement about what part of the attack surface is important/worrisome.
[I absolve you of the tacit duty to respond to this.]