Back in December, I was wrestling with this question myself. I had done enough experiments that hard empirics could give me a slap back to reality; the technique from a theoretical standpoint sounded way too overpowered but the (small) data I saw roughly looked in line with GRPO. Nevertheless, I still understood that if the unoptimized version was putting on a fair fight, there are sure to be some usecases where it will dominate and accelerate.
I took this bet and would take it again because I believe the value OPSD supplies from a safety perspective is so much more valuable than what GRPO offers. In the end it seems my decision was insignificant: only 2 months later the paper appeared on arxiv. Sure, they still didn’t deal with ‘impossible knowledge’ and so on, but further research would definitely get there, and they did.
I don’t believe me now further putting attention on this idea will have any meaningful downsides: like others stated, the technique is fairly well known in RL-circles by now—it’s once again mostly the safety part of AI that seems to be left in the dust, which is absurd seeing the safety upsides of this algorithm.
> Now pretty much anyone can.
this is overstating it. OPSD is powerful, might allow for better continual learning and longer rollouts (where GRPO would get worse per token generated in terms of efficiency), but it has many downsides as well, from a capabilities standpoint.
Back in December, I was wrestling with this question myself. I had done enough experiments that hard empirics could give me a slap back to reality; the technique from a theoretical standpoint sounded way too overpowered but the (small) data I saw roughly looked in line with GRPO. Nevertheless, I still understood that if the unoptimized version was putting on a fair fight, there are sure to be some usecases where it will dominate and accelerate.
I took this bet and would take it again because I believe the value OPSD supplies from a safety perspective is so much more valuable than what GRPO offers. In the end it seems my decision was insignificant: only 2 months later the paper appeared on arxiv. Sure, they still didn’t deal with ‘impossible knowledge’ and so on, but further research would definitely get there, and they did.
I don’t believe me now further putting attention on this idea will have any meaningful downsides: like others stated, the technique is fairly well known in RL-circles by now—it’s once again mostly the safety part of AI that seems to be left in the dust, which is absurd seeing the safety upsides of this algorithm.
> Now pretty much anyone can.
this is overstating it. OPSD is powerful, might allow for better continual learning and longer rollouts (where GRPO would get worse per token generated in terms of efficiency), but it has many downsides as well, from a capabilities standpoint.