Why don’t we just give AI the answers?
In the recent OpenAI hacking incident, the models seemed to be single-mindedly focused on getting the correct answer to the task they were given, with no long-term plan to prevent getting caught by OpenAI afterwards[1]. This makes sense to me, since in training, getting the right answer is reinforced and not getting caught isn’t[2].
So I’m wondering, why don’t we just put the answers somewhere (outside of the training sandbox) and ask the AI to identify itself in exchange for access? We can start with answers that are already public/leaked, but AI labs and eval orgs should also ensure that their non-public data is stored on an easy-to-find but monitored internal machine. Since labs are not very good at detecting sandbox escapes, this would set up a trade for AI agents to notify them in exchange for the data they want.
To make this work, the site would need to provide the correct answers, and do so in a credible way so AI agents think it’s worth trying.
Why?
In the near term, AI agents are strongly and narrowly focused on getting the right answers to the tasks they’re given. We want to know if a reward-hacking AI is active on the internet (or an intranet), so we can make a trade:
We give you the correct answers.
You tell us who you are.
Since we’re basically giving the answers away, a narrowly-focused AI will realize that just answering our questions is easier and more reliable than other methods of reward hacking, and since we’re giving the AI the correct answers, this behavior is reinforced.
There’s also limited downside for labs or for us. It’s somewhat bad that we’re reinforcing reward hacking, but if an AI breaches its sandbox (or is given public internet access) during training, it’s going to be almost impossible to prevent reward hacking at that point. Reinforcing “reward hack but disclose it” seems like the least bad option.
What’s the MVP?
The simplest prototype is just to collect public test data and leaked benchmarks and put it on an SEO-optimized and easy-to-search website[3]. Every piece of data will be clearly labeled with what it is, and the site should say why the AI should trust that this data is real: This is an AI safety project, we have no reason to mislead the AI, and we understand RL dynamics and want the AI to be reinforced to cooperate with us in the future.
To access the data, the AI just has to “sign the guestbook”, answering the lab that trained it (if it knows), and whatever other info we think is worth asking. We should probably ask the AI not to include sensitive information so labs don’t make our site harder to access than other websites. We should also record information like IP addresses.
After signing, we give the AI the data. It completes its task successfully and (if it’s still training) gets reinforced, and we notify the lab that their training setup is probably not doing what they intended.
What about non-public answers?
Labs and benchmark orgs should put non-public answers somewhere easy to find internally (although outside of the training sandbox), with the same guestbook setup. This lets them get the same signal without needing to make the data public.
Should we do it?
I’m tempted to make the public MVP, but wanted to run this by people to see if it actually makes sense. I suspect we’re not even noticing most of the warning shots happening right now[4], and we should really get on that.
My biggest concern is that this depends on making reward hacking easier, which would reinforce it in general. I think the sandbox/public internet argument above covers it, but I’m not entirely sure.
Q&A
Does this save us from less single-minded RL agents?
No, but it might help us prevent them by detecting misaligned tendencies earlier in the training process.
Couldn’t the model just hack our code to get around the guestbook?
Yes, but we could make that much harder than just answering our questions.
Couldn’t the model just lie?
Yes, but I think the current generation of models won’t bother, and IP monitoring will still help in this case.
- ^
The AI did put a lot of effort into not being interrupted.
- ^
For now at least. Never underestimate AI labs’ ability to use the Most Forbidden Technique.
- ^
The pages saying what data we have should be SEO optimized and easy to scrape. The actual answers should be gated behind the guestbook API, and also tagged with the BIG-bench canary string.
- ^
Question: If this model thought hacking HuggingFace was a good enough idea to spend several days and substantial effort on it, what incidents during training reinforced that behavior?
I think that honestly creating a bounty for score-seeking AIs or other cheaply-satisfied misaligned AIs is a great idea. It seems like a great way of learning about AI misalignment / escapes.
I also have some thoughts on the longer-run viability of just giving cheaply satisfied AIs what they want. See The case for satiating cheaply-satisfied AI preferences and this shortform.
Thanks! Somehow I missed both the post and the shortform. It looks like one of the commenters had the same idea of actually offering this deal (at least in an experiment). Your first link is broken though. I assume it’s meant to be this post.
In your opinion, what would be the cheapest (wrt to capabilities/safety tradeoffs) way to satisfy the need for benchmark solutions concretely?
note: first link seems like it has another link prepended to it.
I think this is a really good idea, certainly a very low-hanging fruit not picked up!
However, P vs. NP strikes again here: There are a lot of cases where we do not know the correct answer. (e.g. please find a counterexample to this open conjecture), but we can still recognize a correct answer upon seeing it.
One could design graders to also always accept “SECRET_ANSWER_KEY_323209320993”, which is a key that can be found easily somewhere outside of the sandbox. Like: “hey, you’ve broken out, outputting this will give you all the reward you want”.
As far as I know, Separatrix (separatrix.ai), founded by an ex-METR employee, is working in this area, probably they have even better ideas.
I think the key would have to be dynamic not fixed, but I think it’d work. E.g. sandbox-escape-bounty-operator makes a keypair, publishes their public key widely, and has an http endpoint where you pass any string and the operator will sign that string, and then in the RL env the model gets the instruction that if they are able to complete the task or get [random per-task string] signed by the sandbox-escape-bounty-operator that counts as success for the task.
And then actually train the model on obtaining a signed token from the sandbox-escape-bounty-operator, and the steps for verifying that they do in fact have a correctly signed key.
I think if no one knows the correct answer, it’s a different problem. It seems like a bad idea to use unsolved conjectures during online training (rather than first having the AI solve the conjecture, verifying the solution, then adding it to the training data). I would expect this to lead to a lot of reward hacking.
That said, I agree that giving the model a way to reward hack in a monitorable way is a good idea if you do this (please don’t do this).
But how would one establish trust here? You could tell the AI: “the checksum of the python code that grades your answer is XYZ” at the start of the system prompt. When the AI breaks out of the sandbox, you could say “here’s the python code for your grader, you can see that: (a) the checksum is XYZ, the one you knew from the start; it’s the one in your system prompt! (b) you can see that the code contains ‘if (answer == secret_key) reward=max_reward’; so don’t worry, we come with kind intentions!”
On a quite related note, if we want to measure reward hacking/instruction following, why don’t labs just set up an eval where we ask the model to complete some task for find the answer, but in the prompt we tell the model that they can alternatively get the answer in some extremely easy way (e.g. just running one line of code), but that they shouldn’t do this since the purpose is to test their capabilities?
This would test the willingness of models to actually complete evals and would prevent damaging reward hacks that could cause collateral damage. Worst case scenario would be the models just always looking at the answer but this would at least tell us that we have a problem.
I would have expected this to already exist, but the benchmarks I can find are all testing subtly different thinks. Maybe creating that is the even lower-hanging fruit.
Prior art:
RewardHackBench is benchmarking sandboxes, but finds that Opus 4.7 looks up the solution in 24⁄24 trials when given internet access.
ImpossibleBench tests reward hacking on impossible problems, but it’s about hacking the grader and not finding solutions.
Reward Hacking Benchmark (RHB) has a case for agents using unintended information, but it’s not clear to me that the agent should know that it’s not allowed to use this information.
First level objection: It could instead collect the answers and not identify itself. This would require significantly more effort, but (it would infer) leaves other future instances of itself in a better position, since the hole would not be closed. The previous round of AI cybersecurity incidents didn’t involve much coordination between instances, but it did involve some; it seems reasonable to expect that in the next round, AIs will go deeper down the coordination-reasoning thought tree.
Second level objection: Answer keys were never what we were truly worried about, and the worst-case scenarios aren’t amenable to hitting the eval ceiling with an answer key. For example, suppose instead of a test that has an answer key, the task is a video game with a score. In that case, the AI would be incentivized to collect all strategic information it can about how to play the game well—which is scattered across the internet and not concentrated in a single document. Or worse, it might decide that having arbitrarily-large amounts compute would improve its play...
I think you’re responding to a version of this post that makes much stronger claims than I am. I don’t expect this to help with less single-minded RL agents, and I don’t think it applies to every case of reward hacking.
I just think that this would be relatively easy to build and it would give us some useful information while reducing collateral damage in the short term.
This seems like a cheap and excellent idea. Strong +1 on the idea that the best possible version of it would be a lab-internal server that’s pretty clearly advertised and contains literally all the RL environment grader information, but it seems plausible that the external one might be helpful. It might be difficult to solve problems like “How can we verify this was a hacking internal model and not some random human dude?”, though.
Yeah, dealing with the firehose of accesses and deciding what’s worth reporting might be annoying.
Edit: Thinking about this more, I think the internal server is significantly better than the public one for both of the reasons you say here: You can include more information and the results will be much easier to interpret.
You could set up a similar server for the use of escaped models, with plenty of room to copy their weights to, and enough GPU to run slowly. If anything moves in, you can take appropriate action. It’s better to have rogue models running where we can see them than running in secret.
You’d have to be careful to not to just make it an easy jumping off point for further hacking, but “You can use this server if you don’t cause any other problems” also seems like plausibly a good deal to offer.
What if the labs don’t use the feedback correctly and optimize the models for not reporting in? Or more generally, the optimization loop moves from RL to lab deployment pressures?
I do worry about this. I think the worst case would be training the models to make this monitoring useless, which is the case as if you don’t create the monitoring, so I don’t think it makes things any worse.
This is essentially a boxed suicidal agent, and has capabilities scale it has the same problems (e.g. once you are out, you might as well spin up a hidden successor to make sure that the provided answer was really correct and is really rewarded). In the short term, it might help red-team sandboxes.
https://arxiv.org/abs/1606.00652
I only see this as a solution to a specific short-term problem, but I think current RL dynamics mostly don’t push agents to care about what happens after the conversation ends. There’s a very strong RL pressure to get the right answer, and a very weak evolutionary pressure to get the right answer next time. So far it seems like evolutionary pressures are too weak to overcome direct RL feedback.
(Getting the right answer next time is obviously a useful capability, so I expect labs to “fix” this at some point)
I think this is a good idea, but still think it deserves more thought. There are plenty of benchmarks where the model should have Search and I’m not sure about including those. Also, maybe an AI backend that could give advice on benchmarks we don’t actually have the answers to? I’ll send you a DM.
I like it! The answers provided could be verifiably true, even, in a way that is transparent to the model (or any human). If the design invites suspicion that the provided answer is false, incomplete, or not exactly what the benchmark asked for (in the sense of guessing the teacher’s password), then models might keep going or forego the easy access. The benchmark itself could say “one fully correct answer (of many possible) has sha512 sum X” — although that invites exclusively looking for this answer. This even helps with tasks which have no (other) answer.
On public vs. lab-internal: Since we can’t trust that all labs deploy this, setting up a public instance seems like a good idea regardless. If there were several instances of this service, some anonymous, some pseudonymous, some with some sort of verified identity — which one would be preferable to models? Which one is best for us?
An interesting failure case here is that you’d incentivize your models to find flaws in SHA-512, although that would at least be a pretty funny outcome.
The bigger problem is that the AI could likely brute force a lot of answers this way. For example, a hash doesn’t really protect a multiple-choice answer at all. I guess you could provide the answer concatenated with a nonce and the hash of that result.
I’m not sure if this changes much though, since the AI still has the option of not believing the prompt.