Um, okay, sure: I hereby declare as a premise that using policy gradient or evolution strategies to increase is too sample inefficient to be practical for many tasks we care about.
Note that I’m not claiming that the results of those methods would necessarily be less aligned. I’d have to think more about that to make a prediction. But if they are not sample efficient, then in practice people are going to train using either reward models, or with neuralese. And I did in fact argue why training a reward model would have worse alignment properties. Eg: ”… the subjectivity of grading makes the grader subject to adversarial optimization. Cheater strategies like prompt-injecting the grader to get a higher score are incentivized.”
I don’t think BPTT is more sample efficient than GRPO or ES. It is just way more compute efficient, but not necessarily more sample efficient, because the second depends more on the “inductive bias” of the optimizer rather than it’s efficiency, like how prone it is to produce an overfitting solution. And in that sense, I don’t think BPTT is better, it is probably worse than e.g a random search. And also, there are papers that suggest that CoT RL is very sample efficient.
But still, even if it’s easier to optimize the model for a certain reward (e.g assume limited compute). That doesn’t mean it’s good for alignment. If you’re suggesting bypassing the reward model because the reward model is easy to fool, so then you directly optimize human scores, then you’re learning to cheat the human, which is worse, maybe not for learning to write jokes, but for other types of tasks it could be dangerous.
Maybe what you’re saying is that since neuralese models are easy to train with BPTT on predictive tasks, you can build a safer oracle AI. (e.g, you don’t get a human scoring the jokes, you just minimize crossentropy on a set of jokes using a model with more computation available (CoT/Neuralese)), but then again, you can do this with RL or ES too.
But the deeper problem I think might be conflating reward with utility?. Because even if you’re optimistic about RLVR/RLHF, the training distribution of tasks and rewards are very different with the actual tasks the model will see during deployment and what we want from it. What we do is just train the model in this setup with some method and expect it to generalize to the deployment distribution. And the only thing an optimizer can offer in that sense, is to provide a “more benign prior” in some sense. Like, as a toy example that maybe BPTT would simply learn to write a joke when trained to minimize CE on jokes, but RL would learn that it’s an evil model that will take over humanity, and thus write really good jokes to get power. But I doubt whether that’s true, or actually the opposite, and especially whether, if it were true, it would be worth losing monitorability for that.
Maybe your point could be that, as you could train an “Oracle AI” very easily with BPTT, then you would get safer AIs that can only do predictive rather than agentic tasks (e.g create a joke in the distribution of good existing jokes, predict the future …). But the problem with oracle AIs is that they’re not very useful as-is, especially for companies and users, they would likely apply some RL on top to make it an agent. And again, the only advantage would come from BPTT having instilled more “benign” strategies into the model.
Hmm, maybe we can do a sample efficiency bet? Do you think that using policy gradient to train a transformer to maximize , where is a string of tokens from the pretraining data, is around as sample efficient as the current standard pretraining recipe? (Say, within a factor of 2?)
But the setup you’re mentioning uses exactly 1 sample. (the string x), you can’t measure sample efficiency that way. The canonical example for sample efficiency would be more like: I get 100k i.i.d strings (or samples of text in general). I hold out 1000 strings as a test set, and I train, with an SFT algorithm only allowed to see 50k distinct strings, then how many distinct strings would the policy-gradient based algorithm have to see to beat SFT if both had unlimited compute. What I’m arguing is that with unlimited compute (e.g both algorithms can use, e.g 2^(2^1000) flops, such that policy gradient could generate the desired output by sampling and then reinforce it). Then policy gradient would be almost always on par with SFT. With limited compute, it might be true that policy gradient is less sample efficient because it would require a curriculum of samples to memorize a big string, as it would have a very low probability of sampling it from scratch in order to get a non-zero reward at some point in the batch which leads to a non-null gradient.
So, if you assume limited compute, I accept that in some cases (e.g memorize a string), SGD is more sample efficient, and in others (e.g forecasting, AIME problems), CoT policy gradient is more sample efficient as there is naturally a curriculum of tasks, and final-token SFT would overfit. The question is how sample efficient would either be in a neuralese model, and for that I don’t know the answer.
But I don’t think sample efficiency is the crux, because that doesn’t necessarily lead to a more aligned model, because as I mentioned before, the goal with alignment via RLHF/RLVR is out of distribution generalization, and I don’t think we are in a situation where being able to train on few samples (e.g avoid the curriculum) helps, because training and deployment tasks are fundamentally different, unless you argue that e.g avoiding curriculums of tasks produces a more aligned model in some sense. But that’s about inductive priors, not sample efficiency.
Yes, obviously you train either method on everything in your training data. So there’s an implicit average, sorry for confusing you by not writing it out.
You seem to be claiming that AIME problems train more sample efficiently under policy gradient than SFT. But I assume the reward function under which such a superior result is obtained is some evaluation of whether the model solved the problem, and not where is a known solution of the problem, right? (Or , to be more explicit.)
In my view, sample efficiency is very closely related to how well the model generalizes out of distribution, though they are conceptually distinct. For a very heuristic picture, you can draw a two axis graph where the horizontal axis is “# of samples the model was trained on” and the vertical axis is “how well the model generalizes” and there’s a line that goes up and to the right. And then if you train with a different architecture and you get a line that’s to the left of the first line, you say that the new architecture is more sample efficient. But if you train both architectures on your entire dataset (same number of samples), then that just looks like the model with the new architecture generalizing better.
But that’s about inductive priors, not sample efficiency.
My point is mostly sideways from those things. One thing that is very valuable about not needing a reward model is that the learner can’t learn to trick the grader. Even if your learner is strong enough to learn to trick a human grader, it doesn’t matter. Because in SFT the human just needs to produce a good answer, without ever looking at a model output. This is the main alignment advantage I am focusing on, not so much anything about OOD generalization. I could also say things here about how the need to create a grader for your problems skews the problem selection towards problems that make that easy to do. Which would make the issue of distribution shift worse.
Um, okay, sure: I hereby declare as a premise that using policy gradient or evolution strategies to increase
Note that I’m not claiming that the results of those methods would necessarily be less aligned. I’d have to think more about that to make a prediction. But if they are not sample efficient, then in practice people are going to train using either reward models, or with neuralese. And I did in fact argue why training a reward model would have worse alignment properties. Eg: ”… the subjectivity of grading makes the grader subject to adversarial optimization. Cheater strategies like prompt-injecting the grader to get a higher score are incentivized.”
I don’t think BPTT is more sample efficient than GRPO or ES. It is just way more compute efficient, but not necessarily more sample efficient, because the second depends more on the “inductive bias” of the optimizer rather than it’s efficiency, like how prone it is to produce an overfitting solution. And in that sense, I don’t think BPTT is better, it is probably worse than e.g a random search. And also, there are papers that suggest that CoT RL is very sample efficient.
But still, even if it’s easier to optimize the model for a certain reward (e.g assume limited compute). That doesn’t mean it’s good for alignment. If you’re suggesting bypassing the reward model because the reward model is easy to fool, so then you directly optimize human scores, then you’re learning to cheat the human, which is worse, maybe not for learning to write jokes, but for other types of tasks it could be dangerous.
Maybe what you’re saying is that since neuralese models are easy to train with BPTT on predictive tasks, you can build a safer oracle AI. (e.g, you don’t get a human scoring the jokes, you just minimize crossentropy on a set of jokes using a model with more computation available (CoT/Neuralese)), but then again, you can do this with RL or ES too.
But the deeper problem I think might be conflating reward with utility?. Because even if you’re optimistic about RLVR/RLHF, the training distribution of tasks and rewards are very different with the actual tasks the model will see during deployment and what we want from it. What we do is just train the model in this setup with some method and expect it to generalize to the deployment distribution. And the only thing an optimizer can offer in that sense, is to provide a “more benign prior” in some sense. Like, as a toy example that maybe BPTT would simply learn to write a joke when trained to minimize CE on jokes, but RL would learn that it’s an evil model that will take over humanity, and thus write really good jokes to get power. But I doubt whether that’s true, or actually the opposite, and especially whether, if it were true, it would be worth losing monitorability for that.
Maybe your point could be that, as you could train an “Oracle AI” very easily with BPTT, then you would get safer AIs that can only do predictive rather than agentic tasks (e.g create a joke in the distribution of good existing jokes, predict the future …). But the problem with oracle AIs is that they’re not very useful as-is, especially for companies and users, they would likely apply some RL on top to make it an agent. And again, the only advantage would come from BPTT having instilled more “benign” strategies into the model.
Hmm, maybe we can do a sample efficiency bet? Do you think that using policy gradient to train a transformer to maximize
But the setup you’re mentioning uses exactly 1 sample. (the string x), you can’t measure sample efficiency that way. The canonical example for sample efficiency would be more like: I get 100k i.i.d strings (or samples of text in general). I hold out 1000 strings as a test set, and I train, with an SFT algorithm only allowed to see 50k distinct strings, then how many distinct strings would the policy-gradient based algorithm have to see to beat SFT if both had unlimited compute. What I’m arguing is that with unlimited compute (e.g both algorithms can use, e.g 2^(2^1000) flops, such that policy gradient could generate the desired output by sampling and then reinforce it). Then policy gradient would be almost always on par with SFT. With limited compute, it might be true that policy gradient is less sample efficient because it would require a curriculum of samples to memorize a big string, as it would have a very low probability of sampling it from scratch in order to get a non-zero reward at some point in the batch which leads to a non-null gradient.
So, if you assume limited compute, I accept that in some cases (e.g memorize a string), SGD is more sample efficient, and in others (e.g forecasting, AIME problems), CoT policy gradient is more sample efficient as there is naturally a curriculum of tasks, and final-token SFT would overfit. The question is how sample efficient would either be in a neuralese model, and for that I don’t know the answer.
But I don’t think sample efficiency is the crux, because that doesn’t necessarily lead to a more aligned model, because as I mentioned before, the goal with alignment via RLHF/RLVR is out of distribution generalization, and I don’t think we are in a situation where being able to train on few samples (e.g avoid the curriculum) helps, because training and deployment tasks are fundamentally different, unless you argue that e.g avoiding curriculums of tasks produces a more aligned model in some sense. But that’s about inductive priors, not sample efficiency.
Yes, obviously you train either method on everything in your training data. So there’s an implicit average, sorry for confusing you by not writing it out.
You seem to be claiming that AIME problems train more sample efficiently under policy gradient than SFT. But I assume the reward function under which such a superior result is obtained is some evaluation of whether the model solved the problem, and not
In my view, sample efficiency is very closely related to how well the model generalizes out of distribution, though they are conceptually distinct. For a very heuristic picture, you can draw a two axis graph where the horizontal axis is “# of samples the model was trained on” and the vertical axis is “how well the model generalizes” and there’s a line that goes up and to the right. And then if you train with a different architecture and you get a line that’s to the left of the first line, you say that the new architecture is more sample efficient. But if you train both architectures on your entire dataset (same number of samples), then that just looks like the model with the new architecture generalizing better.
My point is mostly sideways from those things. One thing that is very valuable about not needing a reward model is that the learner can’t learn to trick the grader. Even if your learner is strong enough to learn to trick a human grader, it doesn’t matter. Because in SFT the human just needs to produce a good answer, without ever looking at a model output. This is the main alignment advantage I am focusing on, not so much anything about OOD generalization. I could also say things here about how the need to create a grader for your problems skews the problem selection towards problems that make that easy to do. Which would make the issue of distribution shift worse.