It’s tricky to think about this topic because we don’t know exactly how the AI companies train their LLMs. (Or at least I sure don’t.) But here’s a speculative possibility to explain your puzzle that “the RL capabilities generalize further than the RL misalignment”:
Famously (Ord, Dwarkesh, Beren, etc.), RLVR needs bootstrapping to work, because RLVR doesn’t provide many bits, and there’s no training signal if it never succeeds. The bootstrapping comes from pretraining / SFT.
Likewise, DPO-on-a-hard-task (like refactoring an entire big codebase) needs bootstrapping to work, because DPO doesn’t provide many bits, and the human training signal isn’t very helpful if it never gets anywhere remotely close to refactoring the codebase. And maybe this bootstrapping partly comes from RLVR.
I.e., maybe after RLVR, there’s another expert-DPO stage, and that latter stage gets the model from RLVR-craziness closer to all-things-considered human preferences, at least in DPO-like contexts (which would be more like deployment and less like “graded episodes”). I think this would explain why “the RL capabilities generalize further than the RL misalignment”: the final expert-DPO-on-very-hard-tasks stage can simply directly select for both the capabilities and the (at least superficial) alignment simultaneously.
Now, the mantra that “RL is evil” (Bengio) or “RL is terrifying” (me) or “I’m still scared of RL” (Paul Christiano) is a great default starting point, but it does require some caveats and footnotes. In expert-DPO, there’s a human examining and grading the gestalt of what the LLM has spit out, based on the spirit of their own human intentions. If the LLM optimizes for that, then yes you can get sycophancy or trickery, but if the graders are good and careful, it can be mitigated. The graders can also check the tool call logs for hijinks. Thus DPO is not (necessarily) “evil” in the way that RLVR is. In particular, DPO has some of the safety benefits of “MONA”, in a way that RLVR does not.
If all that is right, is it a sustainable solution to LLM alignment? I feel like my mind goes to my traditional answer: yes, if and only if LLMs never get to ASI. To the extent that this system creates good results, I think it’s load-bearing that both the RLVR and the expert-DPO final stage are mere trickles of bits, not the kind of processes that can build up giant edifices of new superhuman knowledge and capabilities at scale. After all, in the limit, DPO does break down; it relies on the human graders being capable of oversight.
(It’s also possible that the companies go back and forth between expert-DPO and RLVR during training, which would make it even more obvious that we should expect LLMs to wind up with strong behavioral divergence, with ruthlessness in graded-episode contexts and with what-the-DPO-expert-graders-would-want in other contexts.)
It’s tricky to think about this topic because we don’t know exactly how the AI companies train their LLMs. (Or at least I sure don’t.) But here’s a speculative possibility to explain your puzzle that “the RL capabilities generalize further than the RL misalignment”:
Famously (Ord, Dwarkesh, Beren, etc.), RLVR needs bootstrapping to work, because RLVR doesn’t provide many bits, and there’s no training signal if it never succeeds. The bootstrapping comes from pretraining / SFT.
Likewise, DPO-on-a-hard-task (like refactoring an entire big codebase) needs bootstrapping to work, because DPO doesn’t provide many bits, and the human training signal isn’t very helpful if it never gets anywhere remotely close to refactoring the codebase. And maybe this bootstrapping partly comes from RLVR.
I.e., maybe after RLVR, there’s another expert-DPO stage, and that latter stage gets the model from RLVR-craziness closer to all-things-considered human preferences, at least in DPO-like contexts (which would be more like deployment and less like “graded episodes”). I think this would explain why “the RL capabilities generalize further than the RL misalignment”: the final expert-DPO-on-very-hard-tasks stage can simply directly select for both the capabilities and the (at least superficial) alignment simultaneously.
Now, the mantra that “RL is evil” (Bengio) or “RL is terrifying” (me) or “I’m still scared of RL” (Paul Christiano) is a great default starting point, but it does require some caveats and footnotes. In expert-DPO, there’s a human examining and grading the gestalt of what the LLM has spit out, based on the spirit of their own human intentions. If the LLM optimizes for that, then yes you can get sycophancy or trickery, but if the graders are good and careful, it can be mitigated. The graders can also check the tool call logs for hijinks. Thus DPO is not (necessarily) “evil” in the way that RLVR is. In particular, DPO has some of the safety benefits of “MONA”, in a way that RLVR does not.
If all that is right, is it a sustainable solution to LLM alignment? I feel like my mind goes to my traditional answer: yes, if and only if LLMs never get to ASI. To the extent that this system creates good results, I think it’s load-bearing that both the RLVR and the expert-DPO final stage are mere trickles of bits, not the kind of processes that can build up giant edifices of new superhuman knowledge and capabilities at scale. After all, in the limit, DPO does break down; it relies on the human graders being capable of oversight.
(It’s also possible that the companies go back and forth between expert-DPO and RLVR during training, which would make it even more obvious that we should expect LLMs to wind up with strong behavioral divergence, with ruthlessness in graded-episode contexts and with what-the-DPO-expert-graders-would-want in other contexts.)