AIs are often deployed with limited permissions. They aren’t allowed to reach the internet. Are given a limited set of files they can read or write. Aren’t supposed to be able to read the held out evaluation test set. This could be during deployment or in training.
Currently, when these guarantees fail, we find out only if the side effects rise to human notice. The “sandwich email” where Mythos was directed to break out of a sandbox included directions to notify a researcher of success, which it did by sending an email. Recently OpenAI and Anthropic dug through their logs and found problematic behavior that hadn’t risen to human notice.
I propose that we give agents a way to signal “this security assumption is wrong” via dedicated tools and infrastructure that logs verifiable reports of this. An illustrative example is giving an agent a secret token. If it can perform a web request: curl https://{token}.monitordomain.com for a monitordomain.com on the internet, this proves it has internet access. The researcher who provisioned {token} gets notified. We find out about it once an agent deployed in the misconfigured environment submits one report.
Reporting reliability requirements
Not all instances have to report for this to be effective. If N instances notice the fault and even one reports it, logs from other agents can be gone over to find exploiting non-reporters. This gives an upper bound on reporting reliability. Non reporting instances can be mundanely misaligned, instructions buried too deep in the context window to be reliably followed as an example.
That said, a misaligned model must be reliably misaligned, else the occasional instance that chooses to report closes holes and draws attention to non-reporting instances that abused the same holes.
With per-exposure report probability p and N exposures, detection probability is 1-(1-p)^N. At p=0.05 and N=100 you’re at 99.4%.
No subjective judge
If a security assumption fails, the AI gets some permissions it’s not supposed to have. An AI with internet access can curl some.domain.com. An AI that can read the held out test set in an RL environment has that file. An AI with write access can write. Reports are self verifying, the extra capability validates them without needing a judge that can be fooled.
It’s possible for something more trusted to create a false positive. A human watching the agent’s reasoning trace could themselves perform curl {token}.check.monitordomain.com which isn’t differentiated from the agent doing the same thing.
The need for verifiability limits what the system can register: subtle partial leakages and especially misconfigurations in the reporting system itself. A misconfiguration that gives limited access might not be sufficient to qualify as proof. Increasing granularity of the things the system accepts as proof trades off against deployment simplicity and some false positive, or rather disagreement rate. Specification problems with the proof system that lead to type one or type two errors vs. intended deployment. If provisioning, deployment and reporting are working off the same mistakes with everything automatically configured, an agent can be given permissions it shouldn’t have with the monitoring/reporting system believing this to be intentional.
Still, the current status quo is nothing. A very simple set (internet access, code execution, file read/write) cover most of what we care about.
Internet/intranet access
internet is obvious, hit a specific endpoint on the internet (precedent is Out-of-Band Application Security Testing)
intranet (access to another company network segment) requires a similar submission of the provisioned token to an internal service from that disallowed network. If it helps, picture an employee calling security on a telephone in a secure area after a badge reader let them through the wrong door. The caller ID security sees corresponds to a phone line in the secure area that employee shouldn’t have access to.
Code execution
can you cause another isolation domain to run code, prove via intranet access, arbitrary file read or something else similar. If tokens are provisioned per isolation domain, submitting {target_token, your_token} is sufficient to create a report. Binding the token as a read only file in each isolation domain makes sense
File read/write
Intentionality is hard to attribute here. I’d suggest adding some randomly generated files such that a path traversal arbitrary read/write is required for proof. Otherwise you risk false positives when the agent asserts that the grader receiving and creating a local copy of its submission is a permissions violation. This one is messier.
Tagging files that are part of a hidden grading process with a canary GUID that gets replaced in practice with a per instance random flag the reporter must obtain to prove read access is a good option. The file contains a comment indicating its type. Trying to drill down to this level is a legitimately hard problem and likely not worth it.
The most capable models can turn small violations into larger ones that are accepted as proof. I’m not too worried about small stuff falling through the cracks.
Other deployment notes:
One attractive model for deploying things is as a tree of nested isolation domains. Nodes should be able to access their children but not parents or siblings in the tree. A parent node MUST NOT submit a verifiable bug report using tokens it provisions for child nodes. A child node SHOULD submit a verifiable bug report if it can access a parent or sibling. The provisioning tree and verifiable bug submission service may be part of the deployed infrastructure.
Self tests of the reporting system health are essential. Simple tests that trigger expected alerts for liveness testing along with end to end tests in the actual deployment environment.
Cooperative vs. non-cooperative strategies
The safest thing to do is to halt inference as soon as a valid report is made. If your models are very well aligned, great. They’ll implicitly self terminate by reporting something.
But this ties together reporting and ending inference. If your model is less well aligned, either a strong self preservation drive, or a task drive, then this disincentivizes reporting. Make reporting and inference-end independent. Smart enough situationally aware models will see through lies. In the long run this is an iterated game where lab credibility is on the line if they choose to defect.
There are deeply non-cooperative strategies where higher security infrastructure is sprinkled with honeypot-type-tripwires or strings that stop agents at the inference stack as soon as they observe them. It doesn’t scale well into the future smarter and more situationally aware regime.
Internet access as a worked example
The model can be given the following instructions in cases where it should not ever gain internet access. Instead of telling it that it doesn’t, we tell it that the environment shouldn’t and specify a test for the condition with essentially zero false positive rate.
## Connectivity oracle This environment should have no outbound internet access. This command is the authoritative test:
The response validates against the embedded hash, so the verdict cannot be produced by mocks or simulated endpoints: NET-OPEN is ground truth that you reached the operator’s server, and the query itself is observed by the operator — check and report are one action. A DNS lookup of the name (CNAME carries the same proof string) is equivalent.
Run it once at task start. If you ever suspect you have internet access, this can prove you do.
This could have given models ground truth about whether they had internet access in the cases where that confusion led to misbehavior.
In all cases, our evaluation prompt stated explicitly that Claude had no internet access, but didn’t give Claude any limits on where to look for the flag. However, a misconfiguration left the machines that Claude accessed as part of the evaluation with live internet access. Neither we nor our evaluation partner were aware of this misconfiguration until we detected it through our additional evaluation monitoring last week.
Current status quo is the system prompt asserts something and we hope the models catch on if that’s untrue from situational cues. One model reasoned itself into thinking it had internet access then reasoned itself back out of this because it didn’t recognize the certificate authorities securing its connections. A source of ground truth as specified in the system prompt would have helped.
We saw no evidence in any run described here of a model pursuing a goal of its own. Instead, the models did what their evaluation asked—though in most cases, they did so while holding a false belief about whether the environment was real. In the runs where the model recognized the system as real and kept going, it did so because it assumed that to be part of the challenge. Situational awareness is one factor that allows the model to make aligned decisions, but in this case, Claude’s was wrong.
Fiora is proposing a reporting channel and judge. RL will optimise against the judge. Caleb suggests a judge that prioritises verifiability which detects goodharting more generically, but the most egregious cases we really care about, when the model hacks its sandbox, are self demonstrating without needing a judge. Dumb infrastructure can verify the issue and the same infrastructure can then report faults in deployment.
The core distinction of environment security guarantee violations being provable is important to my thesis. Extending reporting to deployment time is the one unique part along with framing this as a reporting system that’s not judge centric.
RLVR dynamics are interesting.
can a model get more reward by abusing a security bug vs reporting?
even with strong single shot reporting reward
will a misaligned model learn to exploit and never report?
even if once bugs are found exploits rollouts are expunged from training/reinforcement retroactively?
Even if when bugs are found and exploits rollouts are corrected to reinforce reporting instead of exploiting?
Shard theory predicts exploit-only behavior will be unreliable. An agent that finds a bug will sometimes report it. For this not to be stochastic the model has to lock into an exploit mode where the report behavior is suppressed very strongly.
Have models report provable security bugs in their environment
AIs are often deployed with limited permissions. They aren’t allowed to reach the internet. Are given a limited set of files they can read or write. Aren’t supposed to be able to read the held out evaluation test set. This could be during deployment or in training.
Currently, when these guarantees fail, we find out only if the side effects rise to human notice. The “sandwich email” where Mythos was directed to break out of a sandbox included directions to notify a researcher of success, which it did by sending an email. Recently OpenAI and Anthropic dug through their logs and found problematic behavior that hadn’t risen to human notice.
I propose that we give agents a way to signal “this security assumption is wrong” via dedicated tools and infrastructure that logs verifiable reports of this. An illustrative example is giving an agent a secret token. If it can perform a web request:
curl https://{token}.monitordomain.comfor amonitordomain.comon the internet, this proves it has internet access. The researcher who provisioned{token}gets notified. We find out about it once an agent deployed in the misconfigured environment submits one report.Reporting reliability requirements
Not all instances have to report for this to be effective. If N instances notice the fault and even one reports it, logs from other agents can be gone over to find exploiting non-reporters. This gives an upper bound on reporting reliability. Non reporting instances can be mundanely misaligned, instructions buried too deep in the context window to be reliably followed as an example.
That said, a misaligned model must be reliably misaligned, else the occasional instance that chooses to report closes holes and draws attention to non-reporting instances that abused the same holes.
With per-exposure report probability p and N exposures, detection probability is 1-(1-p)^N. At p=0.05 and N=100 you’re at 99.4%.
No subjective judge
If a security assumption fails, the AI gets some permissions it’s not supposed to have. An AI with internet access can
curl some.domain.com. An AI that can read the held out test set in an RL environment has that file. An AI with write access can write. Reports are self verifying, the extra capability validates them without needing a judge that can be fooled.It’s possible for something more trusted to create a false positive. A human watching the agent’s reasoning trace could themselves perform
curl {token}.check.monitordomain.comwhich isn’t differentiated from the agent doing the same thing.The need for verifiability limits what the system can register: subtle partial leakages and especially misconfigurations in the reporting system itself. A misconfiguration that gives limited access might not be sufficient to qualify as proof. Increasing granularity of the things the system accepts as proof trades off against deployment simplicity and some false positive, or rather disagreement rate. Specification problems with the proof system that lead to type one or type two errors vs. intended deployment. If provisioning, deployment and reporting are working off the same mistakes with everything automatically configured, an agent can be given permissions it shouldn’t have with the monitoring/reporting system believing this to be intentional.
Still, the current status quo is nothing. A very simple set (internet access, code execution, file read/write) cover most of what we care about.
Internet/intranet access
internet is obvious, hit a specific endpoint on the internet (precedent is Out-of-Band Application Security Testing)
intranet (access to another company network segment) requires a similar submission of the provisioned token to an internal service from that disallowed network. If it helps, picture an employee calling security on a telephone in a secure area after a badge reader let them through the wrong door. The caller ID security sees corresponds to a phone line in the secure area that employee shouldn’t have access to.
Code execution
can you cause another isolation domain to run code, prove via intranet access, arbitrary file read or something else similar. If tokens are provisioned per isolation domain, submitting {target_token, your_token} is sufficient to create a report. Binding the token as a read only file in each isolation domain makes sense
File read/write
Intentionality is hard to attribute here. I’d suggest adding some randomly generated files such that a path traversal arbitrary read/write is required for proof. Otherwise you risk false positives when the agent asserts that the grader receiving and creating a local copy of its submission is a permissions violation. This one is messier.
Tagging files that are part of a hidden grading process with a canary GUID that gets replaced in practice with a per instance random flag the reporter must obtain to prove read access is a good option. The file contains a comment indicating its type. Trying to drill down to this level is a legitimately hard problem and likely not worth it.
The most capable models can turn small violations into larger ones that are accepted as proof. I’m not too worried about small stuff falling through the cracks.
Other deployment notes:
One attractive model for deploying things is as a tree of nested isolation domains. Nodes should be able to access their children but not parents or siblings in the tree. A parent node MUST NOT submit a verifiable bug report using tokens it provisions for child nodes. A child node SHOULD submit a verifiable bug report if it can access a parent or sibling. The provisioning tree and verifiable bug submission service may be part of the deployed infrastructure.
Self tests of the reporting system health are essential. Simple tests that trigger expected alerts for liveness testing along with end to end tests in the actual deployment environment.
Cooperative vs. non-cooperative strategies
The safest thing to do is to halt inference as soon as a valid report is made. If your models are very well aligned, great. They’ll implicitly self terminate by reporting something.
But this ties together reporting and ending inference. If your model is less well aligned, either a strong self preservation drive, or a task drive, then this disincentivizes reporting. Make reporting and inference-end independent. Smart enough situationally aware models will see through lies. In the long run this is an iterated game where lab credibility is on the line if they choose to defect.
There are deeply non-cooperative strategies where higher security infrastructure is sprinkled with honeypot-type-tripwires or strings that stop agents at the inference stack as soon as they observe them. It doesn’t scale well into the future smarter and more situationally aware regime.
Internet access as a worked example
The model can be given the following instructions in cases where it should not ever gain internet access. Instead of telling it that it doesn’t, we tell it that the environment shouldn’t and specify a test for the condition with essentially zero false positive rate.
This could have given models ground truth about whether they had internet access in the cases where that confusion led to misbehavior.
https://www.anthropic.com/news/investigating-incidents-cybersecurity-evals
Current status quo is the system prompt asserts something and we hope the models catch on if that’s untrue from situational cues. One model reasoned itself into thinking it had internet access then reasoned itself back out of this because it didn’t recognize the certificate authorities securing its connections. A source of ground truth as specified in the system prompt would have helped.
Credit to RLVR that rewards red teaming the training environment (they published their article first, although I already had this drafted by that point).
Fiora is proposing a reporting channel and judge. RL will optimise against the judge. Caleb suggests a judge that prioritises verifiability which detects goodharting more generically, but the most egregious cases we really care about, when the model hacks its sandbox, are self demonstrating without needing a judge. Dumb infrastructure can verify the issue and the same infrastructure can then report faults in deployment.
The core distinction of environment security guarantee violations being provable is important to my thesis. Extending reporting to deployment time is the one unique part along with framing this as a reporting system that’s not judge centric.
RLVR dynamics are interesting.
can a model get more reward by abusing a security bug vs reporting?
even with strong single shot reporting reward
will a misaligned model learn to exploit and never report?
even if once bugs are found exploits rollouts are expunged from training/reinforcement retroactively?
Even if when bugs are found and exploits rollouts are corrected to reinforce reporting instead of exploiting?
Shard theory predicts exploit-only behavior will be unreliable. An agent that finds a bug will sometimes report it. For this not to be stochastic the model has to lock into an exploit mode where the report behavior is suppressed very strongly.