Interesting results RE DPO. OLMo was trained on a mixture of off-policy and on-policy DPO. Off policy DPO is well-known to be really bad. Based on how many Anthropic Fellows projects use DPO, I suspect that the internal Anthropic pipelines make at least some use of DPO. My tentative conclusion is that if you use entirely on-policy DPO (or at least entirely close-to-policy DPO) then it probably does work. I hold this belief pretty weakly, however.
By close-to-policy DPO, I mean things which break the literal meaning of on-policy, but not the spirit like:
Sampling a big DPO dataset from the SFT model, then using that dataset all together, rather than resampling mid-training
Using system-prompted or hinted rollouts as the “accepted” rollouts rather than relying entirely on natural variation
Interesting results RE DPO. OLMo was trained on a mixture of off-policy and on-policy DPO. Off policy DPO is well-known to be really bad. Based on how many Anthropic Fellows projects use DPO, I suspect that the internal Anthropic pipelines make at least some use of DPO. My tentative conclusion is that if you use entirely on-policy DPO (or at least entirely close-to-policy DPO) then it probably does work. I hold this belief pretty weakly, however.
By close-to-policy DPO, I mean things which break the literal meaning of on-policy, but not the spirit like:
Sampling a big DPO dataset from the SFT model, then using that dataset all together, rather than resampling mid-training
Using system-prompted or hinted rollouts as the “accepted” rollouts rather than relying entirely on natural variation