Additional thoughts on how RL fine-tuning would induce obfuscation.
Evading process supervision: It’s plausible that we won’t be able to provide outcome-based oversight, but instead rely on process-based oversight. If the overseer penalises wrong reasoning, but is less likely to detect ‘subtle’ or obfuscated versions of this, then LMs are more likely to obfuscate their reasoning.
Reward hacking. Exploiting imperfections in the reward model may cause language models to obfuscate their reasoning. E.g. using more technical jargon often pattern-matches to competence and expertise. Human raters might assign higher scores to responses that use sophisticated-sounding language, even if the underlying reasoning isn’t better (and might be worse).
Currently uncertain about whether this is a good threat model, since both of these concerns are not applicable to math / coding RL fine-tuning (where it’s not possible to obfuscate reasoning, and where the ground-truth verifier is perfect)
Additional thoughts on how RL fine-tuning would induce obfuscation.
Evading process supervision: It’s plausible that we won’t be able to provide outcome-based oversight, but instead rely on process-based oversight. If the overseer penalises wrong reasoning, but is less likely to detect ‘subtle’ or obfuscated versions of this, then LMs are more likely to obfuscate their reasoning.
Reward hacking. Exploiting imperfections in the reward model may cause language models to obfuscate their reasoning. E.g. using more technical jargon often pattern-matches to competence and expertise. Human raters might assign higher scores to responses that use sophisticated-sounding language, even if the underlying reasoning isn’t better (and might be worse).
Currently uncertain about whether this is a good threat model, since both of these concerns are not applicable to math / coding RL fine-tuning (where it’s not possible to obfuscate reasoning, and where the ground-truth verifier is perfect)