I think it is fairly well known; I’ve heard of it and seen it on X for a while. The paper was published in January, the cursor blog post was published more than a year ago, and Dwarkesh Patel even published a video about it. I’m fairly sure the reason K3 used a variant of PPO instead isn’t because their capabilities researchers (whose job it is to keep up with such things) didn’t hear about it, but rather for simplicity/efficiency reasons. It’s also not that much better than the other methods; e.g. see page 7 of this paper.
Have you read section 4.2 (Main Results) of that same paper? In particular:
Notably, OPSD achieves these gains using only a single rollout per problem and converges within 100 steps, with each problem requiring only 1024 sampled tokens, whereas GRPO requires 8 rollouts of 16k tokens each and may exhibit performance degradation in later steps due to entropy collapse.
The correct figure if you want to see sample efficiency in action is Figure 3.
Furthermore, the comparisons in Figure 3 and Table 2 don’t update the OPSD teacher as they train. In another paper I have read (“Reinforcement Learning via Self-Distillation”), the authors demonstrate that if they repeatedly do a rollout and then update the OPSD student (and consequently teacher, since they are the same model), they get significant improvements. (See Figure 10 on page 11. Note that this figure only shows that the teacher improves significantly with bootstrapping, but this seems to me to also indicate that the student would—note how the student breaks through the barrier set by the initial teacher.)
Next, OPSD works significantly better for continual learning than naive GRPO, but I don’t know how much of an advantage that is because there are other methods that work decently.
Finally, OPSD is still a rather new technique, so unlike GRPO, researchers haven’t found all the little tricks that make it work better, and it’s still crushing GRPO. The OPSD technique is also very flexible, you can throw all kinds of extra context at the teacher to help it, and so there is an entire new direction for optimization.
I think OPSD is a very powerful technique. But maybe all the capabilities researchers had already heard of it, if you and Caleb Biddulph are any indication.
I think it is fairly well known; I’ve heard of it and seen it on X for a while. The paper was published in January, the cursor blog post was published more than a year ago, and Dwarkesh Patel even published a video about it. I’m fairly sure the reason K3 used a variant of PPO instead isn’t because their capabilities researchers (whose job it is to keep up with such things) didn’t hear about it, but rather for simplicity/efficiency reasons. It’s also not that much better than the other methods; e.g. see page 7 of this paper.
quick note: the cursor blog post is a few months old (a year ago the idea didn’t even exist yet)
Sorry, you’re right, I misread the date.
Have you read section 4.2 (Main Results) of that same paper? In particular:
The correct figure if you want to see sample efficiency in action is Figure 3.
Furthermore, the comparisons in Figure 3 and Table 2 don’t update the OPSD teacher as they train. In another paper I have read (“Reinforcement Learning via Self-Distillation”), the authors demonstrate that if they repeatedly do a rollout and then update the OPSD student (and consequently teacher, since they are the same model), they get significant improvements. (See Figure 10 on page 11. Note that this figure only shows that the teacher improves significantly with bootstrapping, but this seems to me to also indicate that the student would—note how the student breaks through the barrier set by the initial teacher.)
Next, OPSD works significantly better for continual learning than naive GRPO, but I don’t know how much of an advantage that is because there are other methods that work decently.
Finally, OPSD is still a rather new technique, so unlike GRPO, researchers haven’t found all the little tricks that make it work better, and it’s still crushing GRPO. The OPSD technique is also very flexible, you can throw all kinds of extra context at the teacher to help it, and so there is an entire new direction for optimization.
I think OPSD is a very powerful technique. But maybe all the capabilities researchers had already heard of it, if you and Caleb Biddulph are any indication.