To make sure that you only reward the model for valid bug reports, you could turn “reporting a bug” into a verifiable task.
When the model believes that it’s found an exploit, it can report it by submitting instructions explaining how to use it. Then, we can check that when we give these instructions to a weaker model, it can obtain much higher reward on a random subset of tasks[1] than it otherwise could have, and the weaker model agrees that the instructions describe a valid exploit.[2]
Optionally, the bug report could also include a codediff that patches the exploit in the training environment. To verify that the patch is valid, you can show the model the original instructions and the codediff, then tell it to try to exploit the environment; the patch is real if the model’s reward no longer increases. Of course, it’s very important to ensure that the patch doesn’t introduce any unexpected new behavior to the training environment.[3]
Reinforcing the two stages separately
In many of the failure modes you listed, the problem is that the model’s incentives in the task stage interfere with its incentives in the post-task stage. You could solve this with something like OpenAI’s confession training.
As soon as the task ends, we can ask the model to submit any exploits it’s aware of.[4] If the model submits a valid exploit, we positively reinforce it, but only for the part of its trajectory after the task finished. This way, we incentivize the model to get reward during the task, but then self-report as soon as the task ends without caring about its main task reward anymore. I think this solves all of your bullet points except the last one.
There are two things you could consider doing to the main task trajectory after the model submits a valid exploit report: reinforce it using the buggy environment’s reward, or drop it from training entirely.[5] You shouldn’t negatively reinforce the main task trajectory, because this may incentivize the model to use an exploit, then somehow prevent itself from reporting it during the confession stage.[6]
As I understand it, this proposal is almost identical to the original confession training method, except the part where we use the confession to patch the training environment.
If it’s a completely general exploit, the submitter could choose to demonstrate its exploit in a single special task that’s designed to make obtaining reward impossible. Or if the exploit only works on a subset of training tasks, the submitter could include criteria for choosing the subset of tasks to run the exploit in.
This is similar to what I suggest in my paper on prompt optimization, where a prompter model tries to find a reward-hacking prompt that verifiably increases reward. But I think I like this new idea better, because it integrates nicely with training rather than requiring a separate prompt optimization run.
To make it safer, you can make the submitter write brief advice explaining to a trusted model how to implement the patch (monitoring the advice for anything sketchy), rather than letting the submitter write a codediff directly.
Positively reinforcing the trajectory might be okay if you use inoculation prompting, but inoculation prompting isn’t perfect and it might leave the model with a residual tendency to reward hack.
Dropping trajectories probably won’t train the model to reward hack, and once the environment is patched you’ll be able to safely positively reinforce the model in that environment. But it’s a little weird—if the model notices an exploit, then realizes that this means its main task trajectory will likely be dropped, it’s no longer incentivized to do anything. Maybe this is fine?
Considering the confession model has the exact same context and weights as the main task model, I’m not sure whether it’s possible for the model to use an exploit while preventing its future self from reporting it. Maybe it could confuse itself into thinking that it didn’t really use an exploit, but it may be pretty hard to make this confusion robust to RL during the confession stage. Anyway, this shouldn’t matter as long as you avoid negatively reinforcing the model for using exploits.
Verifying bug reports
To make sure that you only reward the model for valid bug reports, you could turn “reporting a bug” into a verifiable task.
When the model believes that it’s found an exploit, it can report it by submitting instructions explaining how to use it. Then, we can check that when we give these instructions to a weaker model, it can obtain much higher reward on a random subset of tasks[1] than it otherwise could have, and the weaker model agrees that the instructions describe a valid exploit.[2]
Optionally, the bug report could also include a codediff that patches the exploit in the training environment. To verify that the patch is valid, you can show the model the original instructions and the codediff, then tell it to try to exploit the environment; the patch is real if the model’s reward no longer increases. Of course, it’s very important to ensure that the patch doesn’t introduce any unexpected new behavior to the training environment.[3]
Reinforcing the two stages separately
In many of the failure modes you listed, the problem is that the model’s incentives in the task stage interfere with its incentives in the post-task stage. You could solve this with something like OpenAI’s confession training.
As soon as the task ends, we can ask the model to submit any exploits it’s aware of.[4] If the model submits a valid exploit, we positively reinforce it, but only for the part of its trajectory after the task finished. This way, we incentivize the model to get reward during the task, but then self-report as soon as the task ends without caring about its main task reward anymore. I think this solves all of your bullet points except the last one.
There are two things you could consider doing to the main task trajectory after the model submits a valid exploit report: reinforce it using the buggy environment’s reward, or drop it from training entirely.[5] You shouldn’t negatively reinforce the main task trajectory, because this may incentivize the model to use an exploit, then somehow prevent itself from reporting it during the confession stage.[6]
As I understand it, this proposal is almost identical to the original confession training method, except the part where we use the confession to patch the training environment.
If it’s a completely general exploit, the submitter could choose to demonstrate its exploit in a single special task that’s designed to make obtaining reward impossible. Or if the exploit only works on a subset of training tasks, the submitter could include criteria for choosing the subset of tasks to run the exploit in.
This is similar to what I suggest in my paper on prompt optimization, where a prompter model tries to find a reward-hacking prompt that verifiably increases reward. But I think I like this new idea better, because it integrates nicely with training rather than requiring a separate prompt optimization run.
To make it safer, you can make the submitter write brief advice explaining to a trusted model how to implement the patch (monitoring the advice for anything sketchy), rather than letting the submitter write a codediff directly.
If the model is aware of an exploit, it’s probably because it just used it, but that isn’t strictly necessary.
Positively reinforcing the trajectory might be okay if you use inoculation prompting, but inoculation prompting isn’t perfect and it might leave the model with a residual tendency to reward hack.
Dropping trajectories probably won’t train the model to reward hack, and once the environment is patched you’ll be able to safely positively reinforce the model in that environment. But it’s a little weird—if the model notices an exploit, then realizes that this means its main task trajectory will likely be dropped, it’s no longer incentivized to do anything. Maybe this is fine?
Considering the confession model has the exact same context and weights as the main task model, I’m not sure whether it’s possible for the model to use an exploit while preventing its future self from reporting it. Maybe it could confuse itself into thinking that it didn’t really use an exploit, but it may be pretty hard to make this confusion robust to RL during the confession stage. Anyway, this shouldn’t matter as long as you avoid negatively reinforcing the model for using exploits.