But the setup you’re mentioning uses exactly 1 sample. (the string x), you can’t measure sample efficiency that way. The canonical example for sample efficiency would be more like: I get 100k i.i.d strings (or samples of text in general). I hold out 1000 strings as a test set, and I train, with an SFT algorithm only allowed to see 50k distinct strings, then how many distinct strings would the policy-gradient based algorithm have to see to beat SFT if both had unlimited compute. What I’m arguing is that with unlimited compute (e.g both algorithms can use, e.g 2^(2^1000) flops, such that policy gradient could generate the desired output by sampling and then reinforce it). Then policy gradient would be almost always on par with SFT. With limited compute, it might be true that policy gradient is less sample efficient because it would require a curriculum of samples to memorize a big string, as it would have a very low probability of sampling it from scratch in order to get a non-zero reward at some point in the batch which leads to a non-null gradient.
So, if you assume limited compute, I accept that in some cases (e.g memorize a string), SGD is more sample efficient, and in others (e.g forecasting, AIME problems), CoT policy gradient is more sample efficient as there is naturally a curriculum of tasks, and final-token SFT would overfit. The question is how sample efficient would either be in a neuralese model, and for that I don’t know the answer.
But I don’t think sample efficiency is the crux, because that doesn’t necessarily lead to a more aligned model, because as I mentioned before, the goal with alignment via RLHF/RLVR is out of distribution generalization, and I don’t think we are in a situation where being able to train on few samples (e.g avoid the curriculum) helps, because training and deployment tasks are fundamentally different, unless you argue that e.g avoiding curriculums of tasks produces a more aligned model in some sense. But that’s about inductive priors, not sample efficiency.
Yes, obviously you train either method on everything in your training data. So there’s an implicit average, sorry for confusing you by not writing it out.
You seem to be claiming that AIME problems train more sample efficiently under policy gradient than SFT. But I assume the reward function under which such a superior result is obtained is some evaluation of whether the model solved the problem, and not where is a known solution of the problem, right? (Or , to be more explicit.)
In my view, sample efficiency is very closely related to how well the model generalizes out of distribution, though they are conceptually distinct. For a very heuristic picture, you can draw a two axis graph where the horizontal axis is “# of samples the model was trained on” and the vertical axis is “how well the model generalizes” and there’s a line that goes up and to the right. And then if you train with a different architecture and you get a line that’s to the left of the first line, you say that the new architecture is more sample efficient. But if you train both architectures on your entire dataset (same number of samples), then that just looks like the model with the new architecture generalizing better.
But that’s about inductive priors, not sample efficiency.
My point is mostly sideways from those things. One thing that is very valuable about not needing a reward model is that the learner can’t learn to trick the grader. Even if your learner is strong enough to learn to trick a human grader, it doesn’t matter. Because in SFT the human just needs to produce a good answer, without ever looking at a model output. This is the main alignment advantage I am focusing on, not so much anything about OOD generalization. I could also say things here about how the need to create a grader for your problems skews the problem selection towards problems that make that easy to do. Which would make the issue of distribution shift worse.
But the setup you’re mentioning uses exactly 1 sample. (the string x), you can’t measure sample efficiency that way. The canonical example for sample efficiency would be more like: I get 100k i.i.d strings (or samples of text in general). I hold out 1000 strings as a test set, and I train, with an SFT algorithm only allowed to see 50k distinct strings, then how many distinct strings would the policy-gradient based algorithm have to see to beat SFT if both had unlimited compute. What I’m arguing is that with unlimited compute (e.g both algorithms can use, e.g 2^(2^1000) flops, such that policy gradient could generate the desired output by sampling and then reinforce it). Then policy gradient would be almost always on par with SFT. With limited compute, it might be true that policy gradient is less sample efficient because it would require a curriculum of samples to memorize a big string, as it would have a very low probability of sampling it from scratch in order to get a non-zero reward at some point in the batch which leads to a non-null gradient.
So, if you assume limited compute, I accept that in some cases (e.g memorize a string), SGD is more sample efficient, and in others (e.g forecasting, AIME problems), CoT policy gradient is more sample efficient as there is naturally a curriculum of tasks, and final-token SFT would overfit. The question is how sample efficient would either be in a neuralese model, and for that I don’t know the answer.
But I don’t think sample efficiency is the crux, because that doesn’t necessarily lead to a more aligned model, because as I mentioned before, the goal with alignment via RLHF/RLVR is out of distribution generalization, and I don’t think we are in a situation where being able to train on few samples (e.g avoid the curriculum) helps, because training and deployment tasks are fundamentally different, unless you argue that e.g avoiding curriculums of tasks produces a more aligned model in some sense. But that’s about inductive priors, not sample efficiency.
Yes, obviously you train either method on everything in your training data. So there’s an implicit average, sorry for confusing you by not writing it out.
You seem to be claiming that AIME problems train more sample efficiently under policy gradient than SFT. But I assume the reward function under which such a superior result is obtained is some evaluation of whether the model solved the problem, and not
In my view, sample efficiency is very closely related to how well the model generalizes out of distribution, though they are conceptually distinct. For a very heuristic picture, you can draw a two axis graph where the horizontal axis is “# of samples the model was trained on” and the vertical axis is “how well the model generalizes” and there’s a line that goes up and to the right. And then if you train with a different architecture and you get a line that’s to the left of the first line, you say that the new architecture is more sample efficient. But if you train both architectures on your entire dataset (same number of samples), then that just looks like the model with the new architecture generalizing better.
My point is mostly sideways from those things. One thing that is very valuable about not needing a reward model is that the learner can’t learn to trick the grader. Even if your learner is strong enough to learn to trick a human grader, it doesn’t matter. Because in SFT the human just needs to produce a good answer, without ever looking at a model output. This is the main alignment advantage I am focusing on, not so much anything about OOD generalization. I could also say things here about how the need to create a grader for your problems skews the problem selection towards problems that make that easy to do. Which would make the issue of distribution shift worse.