DPO sucks, it’s never worked for me or anyone I know. The official tinker example has results that don’t even work. I’ve tried SimPO, IPO, and many variants and libraries too.
I’ve had the best results with on-policy pairs (a la https://arxiv.org/abs/2404.19733 ), though this is ~ just RL and thus does not address the problems in this post
DPO sucks, it’s never worked for me or anyone I know. The official tinker example has results that don’t even work. I’ve tried SimPO, IPO, and many variants and libraries too.
I’ve had the best results with on-policy pairs (a la https://arxiv.org/abs/2404.19733 ), though this is ~ just RL and thus does not address the problems in this post