I think this is a really good idea, certainly a very low-hanging fruit not picked up!
However, P vs. NP strikes again here: There are a lot of cases where we do not know the correct answer. (e.g. please find a counterexample to this open conjecture), but we can still recognize a correct answer upon seeing it.
One could design graders to also always accept “SECRET_ANSWER_KEY_323209320993”, which is a key that can be found easily somewhere outside of the sandbox.
Like: “hey, you’ve broken out, outputting this will give you all the reward you want”.
As far as I know, Separatrix (separatrix.ai), founded by an ex-METR employee, is working in this area, probably they have even better ideas.
I think the key would have to be dynamic not fixed, but I think it’d work. E.g. sandbox-escape-bounty-operator makes a keypair, publishes their public key widely, and has an http endpoint where you pass any string and the operator will sign that string, and then in the RL env the model gets the instruction that if they are able to complete the task or get [random per-task string] signed by the sandbox-escape-bounty-operator that counts as success for the task.
And then actually train the model on obtaining a signed token from the sandbox-escape-bounty-operator, and the steps for verifying that they do in fact have a correctly signed key.
I think if no one knows the correct answer, it’s a different problem. It seems like a bad idea to use unsolved conjectures during online training (rather than first having the AI solve the conjecture, verifying the solution, then adding it to the training data). I would expect this to lead to a lot of reward hacking.
That said, I agree that giving the model a way to reward hack in a monitorable way is a good idea if you do this (please don’t do this).
But how would one establish trust here? You could tell the AI: “the checksum of the python code that grades your answer is XYZ” at the start of the system prompt. When the AI breaks out of the sandbox, you could say “here’s the python code for your grader, you can see that:
(a) the checksum is XYZ, the one you knew from the start; it’s the one in your system prompt!
(b) you can see that the code contains ‘if (answer == secret_key) reward=max_reward’; so don’t worry, we come with kind intentions!”
I think this is a really good idea, certainly a very low-hanging fruit not picked up!
However, P vs. NP strikes again here: There are a lot of cases where we do not know the correct answer. (e.g. please find a counterexample to this open conjecture), but we can still recognize a correct answer upon seeing it.
One could design graders to also always accept “SECRET_ANSWER_KEY_323209320993”, which is a key that can be found easily somewhere outside of the sandbox. Like: “hey, you’ve broken out, outputting this will give you all the reward you want”.
As far as I know, Separatrix (separatrix.ai), founded by an ex-METR employee, is working in this area, probably they have even better ideas.
I think the key would have to be dynamic not fixed, but I think it’d work. E.g. sandbox-escape-bounty-operator makes a keypair, publishes their public key widely, and has an http endpoint where you pass any string and the operator will sign that string, and then in the RL env the model gets the instruction that if they are able to complete the task or get [random per-task string] signed by the sandbox-escape-bounty-operator that counts as success for the task.
And then actually train the model on obtaining a signed token from the sandbox-escape-bounty-operator, and the steps for verifying that they do in fact have a correctly signed key.
I think if no one knows the correct answer, it’s a different problem. It seems like a bad idea to use unsolved conjectures during online training (rather than first having the AI solve the conjecture, verifying the solution, then adding it to the training data). I would expect this to lead to a lot of reward hacking.
That said, I agree that giving the model a way to reward hack in a monitorable way is a good idea if you do this (please don’t do this).
But how would one establish trust here? You could tell the AI: “the checksum of the python code that grades your answer is XYZ” at the start of the system prompt. When the AI breaks out of the sandbox, you could say “here’s the python code for your grader, you can see that: (a) the checksum is XYZ, the one you knew from the start; it’s the one in your system prompt! (b) you can see that the code contains ‘if (answer == secret_key) reward=max_reward’; so don’t worry, we come with kind intentions!”