Have you read section 4.2 (Main Results) of that same paper? In particular:
Notably, OPSD achieves these gains using only a single rollout per problem and converges within 100 steps, with each problem requiring only 1024 sampled tokens, whereas GRPO requires 8 rollouts of 16k tokens each and may exhibit performance degradation in later steps due to entropy collapse.
The correct figure if you want to see sample efficiency in action is Figure 3.
Furthermore, the comparisons in Figure 3 and Table 2 don’t update the OPSD teacher as they train. In another paper I have read (“Reinforcement Learning via Self-Distillation”), the authors demonstrate that if they repeatedly do a rollout and then update the OPSD student (and consequently teacher, since they are the same model), they get significant improvements. (See Figure 10 on page 11. Note that this figure only shows that the teacher improves significantly with bootstrapping, but this seems to me to also indicate that the student would—note how the student breaks through the barrier set by the initial teacher.)
Next, OPSD works significantly better for continual learning than naive GRPO, but I don’t know how much of an advantage that is because there are other methods that work decently.
Finally, OPSD is still a rather new technique, so unlike GRPO, researchers haven’t found all the little tricks that make it work better, and it’s still crushing GRPO. The OPSD technique is also very flexible, you can throw all kinds of extra context at the teacher to help it, and so there is an entire new direction for optimization.
I think OPSD is a very powerful technique. But maybe all the capabilities researchers had already heard of it, if you and Caleb Biddulph are any indication.
Have you read section 4.2 (Main Results) of that same paper? In particular:
The correct figure if you want to see sample efficiency in action is Figure 3.
Furthermore, the comparisons in Figure 3 and Table 2 don’t update the OPSD teacher as they train. In another paper I have read (“Reinforcement Learning via Self-Distillation”), the authors demonstrate that if they repeatedly do a rollout and then update the OPSD student (and consequently teacher, since they are the same model), they get significant improvements. (See Figure 10 on page 11. Note that this figure only shows that the teacher improves significantly with bootstrapping, but this seems to me to also indicate that the student would—note how the student breaks through the barrier set by the initial teacher.)
Next, OPSD works significantly better for continual learning than naive GRPO, but I don’t know how much of an advantage that is because there are other methods that work decently.
Finally, OPSD is still a rather new technique, so unlike GRPO, researchers haven’t found all the little tricks that make it work better, and it’s still crushing GRPO. The OPSD technique is also very flexible, you can throw all kinds of extra context at the teacher to help it, and so there is an entire new direction for optimization.
I think OPSD is a very powerful technique. But maybe all the capabilities researchers had already heard of it, if you and Caleb Biddulph are any indication.