Adding a KL penalty against a pretrained model when doing RL is basically a more mathematically-elegant version of quantilization. Take a known-to-be-reasonably-safe distribution (the distribution produced by the pre-trained model), and then optimize it in a bounded way, which is exactly the goal of quantilization. Interestingly, the penalty is not usually used out of concern for alignment, but because generated outputs turned to mush without it. So the field ended up effectively doing quantilization entirely by accident, just because it empirically worked better. Of course, because it was an accident, this also means they might stop (or may have already). You can do many stages of RL, with the KL penalty just being WRT the model in the previous stage. So you can diverge more and more from the original distribution by repeatedly diverging a little bit. And if people do this, we again lose the nice alignment properties of quantilization, for all that the outputs remain sensible.
The other alignment idea from the old days that is currently being implemented by accident is myopia. A good way to RL-train models is with a 1-reply horizon. The model replies to the user, or the coding agent completes its current task and waits for further instruction, and that’s the end of the episode. Reward is assigned and that’s it. The next turn of the conversation is a whole new episode. This is myopia (limiting the agent’s time horizon for reward, and thus hopefully preventing it from wanting to influence the world in large ways). It’s also a particularly nice kind of myopia, where the way the human’s actions depend on the agent’s output is ignored as something that does not causally affect the reward signal. I think this is also done mostly by accident: Having long episodes makes credit assignment difficult, so you don’t want the episode to be too long, and the reply boundary is a natural cutting-point. But plausibly people might have noticed that letting the AI optimize over how users reacted to its outputs resulted in bad things happening, and decided not to do that? Alternately, maybe this info is out of date, and labs do just train with episodes spanning multiple conversation turns now?
Adding a KL penalty against a pretrained model when doing RL is basically a more mathematically-elegant version of quantilization. Take a known-to-be-reasonably-safe distribution (the distribution produced by the pre-trained model), and then optimize it in a bounded way, which is exactly the goal of quantilization. Interestingly, the penalty is not usually used out of concern for alignment, but because generated outputs turned to mush without it. So the field ended up effectively doing quantilization entirely by accident, just because it empirically worked better. Of course, because it was an accident, this also means they might stop (or may have already). You can do many stages of RL, with the KL penalty just being WRT the model in the previous stage. So you can diverge more and more from the original distribution by repeatedly diverging a little bit. And if people do this, we again lose the nice alignment properties of quantilization, for all that the outputs remain sensible.
The other alignment idea from the old days that is currently being implemented by accident is myopia. A good way to RL-train models is with a 1-reply horizon. The model replies to the user, or the coding agent completes its current task and waits for further instruction, and that’s the end of the episode. Reward is assigned and that’s it. The next turn of the conversation is a whole new episode. This is myopia (limiting the agent’s time horizon for reward, and thus hopefully preventing it from wanting to influence the world in large ways). It’s also a particularly nice kind of myopia, where the way the human’s actions depend on the agent’s output is ignored as something that does not causally affect the reward signal. I think this is also done mostly by accident: Having long episodes makes credit assignment difficult, so you don’t want the episode to be too long, and the reply boundary is a natural cutting-point. But plausibly people might have noticed that letting the AI optimize over how users reacted to its outputs resulted in bad things happening, and decided not to do that? Alternately, maybe this info is out of date, and labs do just train with episodes spanning multiple conversation turns now?