All targets which one might use for reinforcement learning seem to be subject to Goodhart’s Law, in a sense. If we treat them as an imperfect measures of “alignment”, then the “misalignment” we see is simply the difference (in concept-space) between the measure (as far as it can actually be faithfully implemented) and alignment:
The difference between “user approved” and “aligned with actual interests” is “user approved but not actually aligned with their interest”, i.e. sycophancy, and that’s the result of doing RLHF;
The difference between “passes the tests” and “aligned with actual interests” is “tests pass without actually having solved the exercise”, i.e. cheating (whether by manipulating the tests or stealing the answer sheet), and that’s the result of doing RLVR;
The difference between “judged correct by peer” and “aligned with actual interests” is “provides convincing, hard to verify, apparently-successful outputs”, i.e. the weird “slippery” outputs @ryan_greenblatt alludes to that result from RLAIF.
If you use more than one of these RL techniques, you get a mixture of the results, eliciting whatever seems to fit the presumed grader best.
So far, I hope I have understood you correctly.
But then, what’s the solution? Reinforcement learning on whatever best measure of “alignment” we have, that is, reduce the difference between model behavior and our best measure of perfect alignment / Yudkowskian “Coherent Extrapolated Volition”? Well, that would necessitate that we have a theory of alignment and could quantify and measure it with high fidelity. And, well, we don’t seem to have that.
But in a sense, we do have something like it: I see all of the above failure modes in children and particularly in students. There might, therefore, be something to learn from pedagogical sciences on how you help children grow up to be broadly aligned members of society (which arguably sometimes works), despite not having a rigorous theory of what “the good” is in humans. How does one grow a good human? I suspect it’s murky and benefits from young humans being malleable and not perfectly ruthless responders to optimization pressure.
Might we go back to imitative learning on already-grown humans, then — the very narrow subset of the best-aligned humans we know of? Basically curate the dataset further and further, until the failure modes disappear? Or do we have to use the mechanisms we know are baked into humans (is this the “brain-like AGI” agenda)?
I suspect that RL with a more thorough reward could help. When a real-world human tries to present slop, slop takes a rather long time to be revealed to be slop and punished accordingly. What the LLM has is a context window which is graded according to the reward model. What if we ask the reward model to be a just-as-capable LLM and to use the code in the project, then to report all the failures to the instance which generated the code and have the instance finetuned on the user’s rant?
All targets which one might use for reinforcement learning seem to be subject to Goodhart’s Law, in a sense. If we treat them as an imperfect measures of “alignment”, then the “misalignment” we see is simply the difference (in concept-space) between the measure (as far as it can actually be faithfully implemented) and alignment:
The difference between “user approved” and “aligned with actual interests” is “user approved but not actually aligned with their interest”, i.e. sycophancy, and that’s the result of doing RLHF;
The difference between “passes the tests” and “aligned with actual interests” is “tests pass without actually having solved the exercise”, i.e. cheating (whether by manipulating the tests or stealing the answer sheet), and that’s the result of doing RLVR;
The difference between “judged correct by peer” and “aligned with actual interests” is “provides convincing, hard to verify, apparently-successful outputs”, i.e. the weird “slippery” outputs @ryan_greenblatt alludes to that result from RLAIF.
If you use more than one of these RL techniques, you get a mixture of the results, eliciting whatever seems to fit the presumed grader best.
So far, I hope I have understood you correctly.
But then, what’s the solution? Reinforcement learning on whatever best measure of “alignment” we have, that is, reduce the difference between model behavior and our best measure of perfect alignment / Yudkowskian “Coherent Extrapolated Volition”? Well, that would necessitate that we have a theory of alignment and could quantify and measure it with high fidelity. And, well, we don’t seem to have that.
But in a sense, we do have something like it: I see all of the above failure modes in children and particularly in students. There might, therefore, be something to learn from pedagogical sciences on how you help children grow up to be broadly aligned members of society (which arguably sometimes works), despite not having a rigorous theory of what “the good” is in humans. How does one grow a good human? I suspect it’s murky and benefits from young humans being malleable and not perfectly ruthless responders to optimization pressure.
Might we go back to imitative learning on already-grown humans, then — the very narrow subset of the best-aligned humans we know of? Basically curate the dataset further and further, until the failure modes disappear? Or do we have to use the mechanisms we know are baked into humans (is this the “brain-like AGI” agenda)?
I suspect that RL with a more thorough reward could help. When a real-world human tries to present slop, slop takes a rather long time to be revealed to be slop and punished accordingly. What the LLM has is a context window which is graded according to the reward model. What if we ask the reward model to be a just-as-capable LLM and to use the code in the project, then to report all the failures to the instance which generated the code and have the instance finetuned on the user’s rant?