In contrast, the training signal from RL will always, ultimately, point towards grader hacking, due to (in-practice irreducible) error in specifying rewards.
Always? It seems possible to me that this is a contingent feature of the world and one’s RL environments rather than a necessary feature of one’s RL environments?
Like—it’s possible to train a human teenager on a test, and give the human teenager a notion of why they’re taking a test, such that when the teenager discovers a problem in the test they’re like “Yo, there’s a problem with the test, let’s fix this.” This might require a bit of pedagogical knowledge or psychology but I don’t think this is impossible. So it’s possible to have a human education, where the human being educated doesn’t try to do grader hacking.
Of course, one might say that some of the many differences between policy gradient and human education make this impossible. But I’m not sure which specific difference it is which promotes this. A base model surely can have an intention at some point (early on?) in RL to avoid grader hacking—this is within the space of possible personas! And an RL environment reward function seems like it could have an affordance that reinforces the intention to not do grader hacking. So I’m not sure that, in principle, it must be necessary for the training signal from RL to always point to grader hacking, given all possible interventions over {RL environments reinforcing anti-hacking intentions, reward-function randomization, more careful user-modeling, arbitrary degrees of per-episode LLM oversight to check for reward-hacking outcomes} and so on.
Like, maybe it is impossible, and the training signal from RL does always point to grader hacking—but it seems like there’s tons of stuff here that has barely been checked here, and tons of knowledge that yet remains to be found, idk.
I do like the human analogy. It’s plausible there’s something there. This is kind of what the Resolution blogpost is getting at—by starting RL from a benign initialization, we hope that this incentivises an honest training generalization. But it is very much a hope at the moment, and too speculative to really base any safety assurances on. Like, this whole agenda might just be too difficult to make progress on (though I do think it’s valuable to have evidence of hardness, in that case).
Re: how RL differs from human education, my sense is that the ~main bottleneck is credit assignment. With human education, it’s possible to decouple motivation from behaviour, and reward good motivations that have bad outcomes. E.g. if a student does all the “right” things but fails to achieve success, a human teacher would typically encourage them to keep trying. More importantly, it’s usually possible to disincentivise bad motivations that have good outcomes—e.g. cheating on a test.
This is not how current LLM RL techniques work. The SOTA techniques reported by labs (including GRPO, RLVR, and their variants) use trajectory-level rewards that are determined entirely by the outcome, not the process. IMO this means that it is ~impossible to reward good motivations separately from good behaviour.
In game theory terms, I think being honest (as a strategy) is dominated by grader hacking under the typical RL objective, in the sense that both strategies would achieve the same reward assuming a perfectly designed RL env. As a result grader hacking can never be disincentivised. Furthermore a very small of misspecified error should be sufficient to push the model to generalise towards grader hacking rather than being honest.
I think there are promising approaches to start from a benign AI initialization and optimize it to improve its performance without teaching it crazy reward-seeking behaviors. I’m working on LLM optimization techniques that hopefully maintain alignment even with an imperfect objective function. See this comment:
My latest research is motivated by the idea that if you have an aligned AI, you can try to improve its performance while still keeping it similar to its old self. This way, we avoid teaching the AI crazy misaligned behaviors. One way to keep the AI from changing too much is to make sure that all updates are endorsed by the original AI [...] Another way is to limit the size of the update or the amount of information you send to improve the AI, which is what I’ve been focusing on so far.
Always? It seems possible to me that this is a contingent feature of the world and one’s RL environments rather than a necessary feature of one’s RL environments?
Like—it’s possible to train a human teenager on a test, and give the human teenager a notion of why they’re taking a test, such that when the teenager discovers a problem in the test they’re like “Yo, there’s a problem with the test, let’s fix this.” This might require a bit of pedagogical knowledge or psychology but I don’t think this is impossible. So it’s possible to have a human education, where the human being educated doesn’t try to do grader hacking.
Of course, one might say that some of the many differences between policy gradient and human education make this impossible. But I’m not sure which specific difference it is which promotes this. A base model surely can have an intention at some point (early on?) in RL to avoid grader hacking—this is within the space of possible personas! And an RL environment reward function seems like it could have an affordance that reinforces the intention to not do grader hacking. So I’m not sure that, in principle, it must be necessary for the training signal from RL to always point to grader hacking, given all possible interventions over {RL environments reinforcing anti-hacking intentions, reward-function randomization, more careful user-modeling, arbitrary degrees of per-episode LLM oversight to check for reward-hacking outcomes} and so on.
Like, maybe it is impossible, and the training signal from RL does always point to grader hacking—but it seems like there’s tons of stuff here that has barely been checked here, and tons of knowledge that yet remains to be found, idk.
I do like the human analogy. It’s plausible there’s something there. This is kind of what the Resolution blogpost is getting at—by starting RL from a benign initialization, we hope that this incentivises an honest training generalization. But it is very much a hope at the moment, and too speculative to really base any safety assurances on. Like, this whole agenda might just be too difficult to make progress on (though I do think it’s valuable to have evidence of hardness, in that case).
Re: how RL differs from human education, my sense is that the ~main bottleneck is credit assignment. With human education, it’s possible to decouple motivation from behaviour, and reward good motivations that have bad outcomes. E.g. if a student does all the “right” things but fails to achieve success, a human teacher would typically encourage them to keep trying. More importantly, it’s usually possible to disincentivise bad motivations that have good outcomes—e.g. cheating on a test.
This is not how current LLM RL techniques work. The SOTA techniques reported by labs (including GRPO, RLVR, and their variants) use trajectory-level rewards that are determined entirely by the outcome, not the process. IMO this means that it is ~impossible to reward good motivations separately from good behaviour.
In game theory terms, I think being honest (as a strategy) is dominated by grader hacking under the typical RL objective, in the sense that both strategies would achieve the same reward assuming a perfectly designed RL env. As a result grader hacking can never be disincentivised. Furthermore a very small of misspecified error should be sufficient to push the model to generalise towards grader hacking rather than being honest.
I think there are promising approaches to start from a benign AI initialization and optimize it to improve its performance without teaching it crazy reward-seeking behaviors. I’m working on LLM optimization techniques that hopefully maintain alignment even with an imperfect objective function. See this comment:
Expanding on my other comment: I found this recently, which I think neatly summarises reinforcement signals from “social justice”: https://www.lesswrong.com/posts/YRgMCXMbkKBZgMz4M/asymmetric-justice
You get punished for things with bad outcomes
You get rewarded for things with good outcomes and good motivations
You get ~0 credit for doing things with good outcomes but selfish / otherwise bad motivations