Yes, obviously you train either method on everything in your training data. So there’s an implicit average, sorry for confusing you by not writing it out.
You seem to be claiming that AIME problems train more sample efficiently under policy gradient than SFT. But I assume the reward function under which such a superior result is obtained is some evaluation of whether the model solved the problem, and not where is a known solution of the problem, right? (Or , to be more explicit.)
In my view, sample efficiency is very closely related to how well the model generalizes out of distribution, though they are conceptually distinct. For a very heuristic picture, you can draw a two axis graph where the horizontal axis is “# of samples the model was trained on” and the vertical axis is “how well the model generalizes” and there’s a line that goes up and to the right. And then if you train with a different architecture and you get a line that’s to the left of the first line, you say that the new architecture is more sample efficient. But if you train both architectures on your entire dataset (same number of samples), then that just looks like the model with the new architecture generalizing better.
But that’s about inductive priors, not sample efficiency.
My point is mostly sideways from those things. One thing that is very valuable about not needing a reward model is that the learner can’t learn to trick the grader. Even if your learner is strong enough to learn to trick a human grader, it doesn’t matter. Because in SFT the human just needs to produce a good answer, without ever looking at a model output. This is the main alignment advantage I am focusing on, not so much anything about OOD generalization. I could also say things here about how the need to create a grader for your problems skews the problem selection towards problems that make that easy to do. Which would make the issue of distribution shift worse.
Yes, obviously you train either method on everything in your training data. So there’s an implicit average, sorry for confusing you by not writing it out.
You seem to be claiming that AIME problems train more sample efficiently under policy gradient than SFT. But I assume the reward function under which such a superior result is obtained is some evaluation of whether the model solved the problem, and not
In my view, sample efficiency is very closely related to how well the model generalizes out of distribution, though they are conceptually distinct. For a very heuristic picture, you can draw a two axis graph where the horizontal axis is “# of samples the model was trained on” and the vertical axis is “how well the model generalizes” and there’s a line that goes up and to the right. And then if you train with a different architecture and you get a line that’s to the left of the first line, you say that the new architecture is more sample efficient. But if you train both architectures on your entire dataset (same number of samples), then that just looks like the model with the new architecture generalizing better.
My point is mostly sideways from those things. One thing that is very valuable about not needing a reward model is that the learner can’t learn to trick the grader. Even if your learner is strong enough to learn to trick a human grader, it doesn’t matter. Because in SFT the human just needs to produce a good answer, without ever looking at a model output. This is the main alignment advantage I am focusing on, not so much anything about OOD generalization. I could also say things here about how the need to create a grader for your problems skews the problem selection towards problems that make that easy to do. Which would make the issue of distribution shift worse.