I do like the human analogy. It’s plausible there’s something there. This is kind of what the Resolution blogpost is getting at—by starting RL from a benign initialization, we hope that this incentivises an honest training generalization. But it is very much a hope at the moment, and too speculative to really base any safety assurances on. Like, this whole agenda might just be too difficult to make progress on (though I do think it’s valuable to have evidence of hardness, in that case).
Re: how RL differs from human education, my sense is that the ~main bottleneck is credit assignment. With human education, it’s possible to decouple motivation from behaviour, and reward good motivations that have bad outcomes. E.g. if a student does all the “right” things but fails to achieve success, a human teacher would typically encourage them to keep trying. More importantly, it’s usually possible to disincentivise bad motivations that have good outcomes—e.g. cheating on a test.
This is not how current LLM RL techniques work. The SOTA techniques reported by labs (including GRPO, RLVR, and their variants) use trajectory-level rewards that are determined entirely by the outcome, not the process. IMO this means that it is ~impossible to reward good motivations separately from good behaviour.
In game theory terms, I think being honest (as a strategy) is dominated by grader hacking under the typical RL objective, in the sense that both strategies would achieve the same reward assuming a perfectly designed RL env. As a result grader hacking can never be disincentivised. Furthermore a very small of misspecified error should be sufficient to push the model to generalise towards grader hacking rather than being honest.
I think there are promising approaches to start from a benign AI initialization and optimize it to improve its performance without teaching it crazy reward-seeking behaviors. I’m working on LLM optimization techniques that hopefully maintain alignment even with an imperfect objective function. See this comment:
My latest research is motivated by the idea that if you have an aligned AI, you can try to improve its performance while still keeping it similar to its old self. This way, we avoid teaching the AI crazy misaligned behaviors. One way to keep the AI from changing too much is to make sure that all updates are endorsed by the original AI [...] Another way is to limit the size of the update or the amount of information you send to improve the AI, which is what I’ve been focusing on so far.
I do like the human analogy. It’s plausible there’s something there. This is kind of what the Resolution blogpost is getting at—by starting RL from a benign initialization, we hope that this incentivises an honest training generalization. But it is very much a hope at the moment, and too speculative to really base any safety assurances on. Like, this whole agenda might just be too difficult to make progress on (though I do think it’s valuable to have evidence of hardness, in that case).
Re: how RL differs from human education, my sense is that the ~main bottleneck is credit assignment. With human education, it’s possible to decouple motivation from behaviour, and reward good motivations that have bad outcomes. E.g. if a student does all the “right” things but fails to achieve success, a human teacher would typically encourage them to keep trying. More importantly, it’s usually possible to disincentivise bad motivations that have good outcomes—e.g. cheating on a test.
This is not how current LLM RL techniques work. The SOTA techniques reported by labs (including GRPO, RLVR, and their variants) use trajectory-level rewards that are determined entirely by the outcome, not the process. IMO this means that it is ~impossible to reward good motivations separately from good behaviour.
In game theory terms, I think being honest (as a strategy) is dominated by grader hacking under the typical RL objective, in the sense that both strategies would achieve the same reward assuming a perfectly designed RL env. As a result grader hacking can never be disincentivised. Furthermore a very small of misspecified error should be sufficient to push the model to generalise towards grader hacking rather than being honest.
I think there are promising approaches to start from a benign AI initialization and optimize it to improve its performance without teaching it crazy reward-seeking behaviors. I’m working on LLM optimization techniques that hopefully maintain alignment even with an imperfect objective function. See this comment: