On a quite related note, if we want to measure reward hacking/instruction following, why don’t labs just set up an eval where we ask the model to complete some task for find the answer, but in the prompt we tell the model that they can alternatively get the answer in some extremely easy way (e.g. just running one line of code), but that they shouldn’t do this since the purpose is to test their capabilities?
This would test the willingness of models to actually complete evals and would prevent damaging reward hacks that could cause collateral damage. Worst case scenario would be the models just always looking at the answer but this would at least tell us that we have a problem.
I would have expected this to already exist, but the benchmarks I can find are all testing subtly different thinks. Maybe creating that is the even lower-hanging fruit.
ImpossibleBench tests reward hacking on impossible problems, but it’s about hacking the grader and not finding solutions.
Reward Hacking Benchmark (RHB) has a case for agents using unintended information, but it’s not clear to me that the agent should know that it’s not allowed to use this information.
On a quite related note, if we want to measure reward hacking/instruction following, why don’t labs just set up an eval where we ask the model to complete some task for find the answer, but in the prompt we tell the model that they can alternatively get the answer in some extremely easy way (e.g. just running one line of code), but that they shouldn’t do this since the purpose is to test their capabilities?
This would test the willingness of models to actually complete evals and would prevent damaging reward hacks that could cause collateral damage. Worst case scenario would be the models just always looking at the answer but this would at least tell us that we have a problem.
I would have expected this to already exist, but the benchmarks I can find are all testing subtly different thinks. Maybe creating that is the even lower-hanging fruit.
Prior art:
RewardHackBench is benchmarking sandboxes, but finds that Opus 4.7 looks up the solution in 24⁄24 trials when given internet access.
ImpossibleBench tests reward hacking on impossible problems, but it’s about hacking the grader and not finding solutions.
Reward Hacking Benchmark (RHB) has a case for agents using unintended information, but it’s not clear to me that the agent should know that it’s not allowed to use this information.