1) Judge everything including the entire CoT: detailed, powerful though expensive, but also a clear case of The Most Forbidden Technique: you should expect this to induce obfuscated CoT and/or jailbreaking of the judge. So probably unwise to use this as a reward model.
In general, if this finds reward hacking, fixing the reasoning environment and restarting the training run from the checkpoint before this occurred seems like a much better solution that just penalizing the model: that way the model can’t learn to obfuscate (or more accurately, it needs to learn to obfuscate first, when it has no motivation to, before learning to reward hack).
2) Just look at the solution output, and have the reward model judge assess that. This is basically attempting to use a judge to look for reward hacking, or other forms of sloppiness. I suspect labs may already doing this, and starting with the Constitution rather then a detailed rubric, while general, may not be the most efficient approach. Still, seems worth trying.
There seem to be two approaches for this:
1) Judge everything including the entire CoT: detailed, powerful though expensive, but also a clear case of The Most Forbidden Technique: you should expect this to induce obfuscated CoT and/or jailbreaking of the judge. So probably unwise to use this as a reward model.
In general, if this finds reward hacking, fixing the reasoning environment and restarting the training run from the checkpoint before this occurred seems like a much better solution that just penalizing the model: that way the model can’t learn to obfuscate (or more accurately, it needs to learn to obfuscate first, when it has no motivation to, before learning to reward hack).
2) Just look at the solution output, and have the reward model judge assess that. This is basically attempting to use a judge to look for reward hacking, or other forms of sloppiness. I suspect labs may already doing this, and starting with the Constitution rather then a detailed rubric, while general, may not be the most efficient approach. Still, seems worth trying.