Thanks for the summary. I think I originally had two hypotheses:
Models are being allowed to adopt non-greedy strategies for RL due to some outer-loop setup, and an environment which favors a make-mistakes-on-purpose-to-fix-them-later strategy (mistake seeding).
Models are somehow cheating when the same model is also the judge/reward estimator in an RL setup
But I was struggling with the logic for the second one, and how that could specifically produce mistake-seeding behaviour. I couldn’t figure it out. It seemed to me like using the model as its own judge could cause pseudo-random unwanted drift in values, and allow bad behaviour to slip through, but I didn’t see why it should produce mistake seeding or other more intelligent forms of cheating without some kind of outer loop.
The reason I think the first hypothesis also makes sense with AI industry timing is that there’s been a huge push toward synthetic training data, and I think synthetic prompts are one vehicle for outer loops appearing.
Thanks for the summary. I think I originally had two hypotheses:
Models are being allowed to adopt non-greedy strategies for RL due to some outer-loop setup, and an environment which favors a make-mistakes-on-purpose-to-fix-them-later strategy (mistake seeding).
Models are somehow cheating when the same model is also the judge/reward estimator in an RL setup
But I was struggling with the logic for the second one, and how that could specifically produce mistake-seeding behaviour. I couldn’t figure it out. It seemed to me like using the model as its own judge could cause pseudo-random unwanted drift in values, and allow bad behaviour to slip through, but I didn’t see why it should produce mistake seeding or other more intelligent forms of cheating without some kind of outer loop.
The reason I think the first hypothesis also makes sense with AI industry timing is that there’s been a huge push toward synthetic training data, and I think synthetic prompts are one vehicle for outer loops appearing.
I do agree that a weak-minded judge can easily be tricked, and that this could make problems happening in RL worse. This is why I recently advocated for paying high IQ people to do RLHF.