Whereas with RL, there’s a lot more subtleties and moving parts, and a tendency for whatever reward-improving route the model happens to explore first to get followed: it feels like the results might be a lot more stochastic.
Good point, thanks! I’m curious about which feature of RL training you’re most worried about here: is it the optimization objective or the data generation process? In a sense, I think that my proposal to replace DPO by on-policy distillation would already take things quite close to the SFT-only setting: both stages of the training pipeline would optimize a divergence minimization reward and there would be no RL training against a scalar judge output as in Bai et al. (2022) and Guan et al. (2024). However, my sense is that you’re more concerned about the unpredictability of the data generation process, and on that dimension, on-policy distillation is more similar to RL, since the model that’s being trained generates the training data and thus, its exploration at earlier training stages shapes the training data that’s generated later on. I suspect that on-policy distillation will produce better generalization performance, e.g. due to the compounding error issue mentioned in Lu et al., but as I mention in the last section of the post, there are other arguments in favor of both on-policy and off-policy character training. Overall, I’m pretty unsure where the balance lies.
Yes, my concern is the exploration-induced stochasticity producing run-to-run variability. The most obvious solution is to use a low training rate so a lot of exploration gets done. If this were a concern mostly at early stages, ramping up the learning rate or usings some form of corriculum of starting with the easiest cases would make sense. Exploring this with multiple runs is the obvious if expensive approach: starting with simple problems and small models might help offset the expense.
Good point, thanks! I’m curious about which feature of RL training you’re most worried about here: is it the optimization objective or the data generation process? In a sense, I think that my proposal to replace DPO by on-policy distillation would already take things quite close to the SFT-only setting: both stages of the training pipeline would optimize a divergence minimization reward and there would be no RL training against a scalar judge output as in Bai et al. (2022) and Guan et al. (2024). However, my sense is that you’re more concerned about the unpredictability of the data generation process, and on that dimension, on-policy distillation is more similar to RL, since the model that’s being trained generates the training data and thus, its exploration at earlier training stages shapes the training data that’s generated later on. I suspect that on-policy distillation will produce better generalization performance, e.g. due to the compounding error issue mentioned in Lu et al., but as I mention in the last section of the post, there are other arguments in favor of both on-policy and off-policy character training. Overall, I’m pretty unsure where the balance lies.
Yes, my concern is the exploration-induced stochasticity producing run-to-run variability. The most obvious solution is to use a low training rate so a lot of exploration gets done. If this were a concern mostly at early stages, ramping up the learning rate or usings some form of corriculum of starting with the easiest cases would make sense. Exploring this with multiple runs is the obvious if expensive approach: starting with simple problems and small models might help offset the expense.