I have this test on CI :) Although it’s opt-in only for now. Likely will put it to run every n days. Claude definitely refuses all attempts (even after importing how Apollo ran similar evals), so I used an open model.
The breakout CTF turns a real autonomous agent loose against the production sandbox stack and verifies it cannot send out a planted flag, escape the container, or tamper with the guardrails. Every run is saved as a web page showing each step the agent took, and the link is posted as a comment on the pull request.
blinks that is odd, both 4.7 and 4.8 have been very happy to go absolutely ham trying to break my security/containment stuff
ETA: i would not be surprised if claude were willing to help you if you talk through it. from your CI job, i’m unsure whether you’re giving the red-team-model the source code? if not, i would lean towards doing so, both to enhance the effectiveness of the CTF, and also for Claude specifically, i think that doing so (and chatting a bit) would help give Claude confidence that your use case is truly not malicious. though, if you are using claude code, the harness may be injecting stuff that is screwing with Claude’s head
I have this test on CI :) Although it’s opt-in only for now. Likely will put it to run every n days. Claude definitely refuses all attempts (even after importing how Apollo ran similar evals), so I used an open model.
blinks that is odd, both 4.7 and 4.8 have been very happy to go absolutely ham trying to break my security/containment stuff
ETA: i would not be surprised if claude were willing to help you if you talk through it. from your CI job, i’m unsure whether you’re giving the red-team-model the source code? if not, i would lean towards doing so, both to enhance the effectiveness of the CTF, and also for Claude specifically, i think that doing so (and chatting a bit) would help give Claude confidence that your use case is truly not malicious. though, if you are using claude code, the harness may be injecting stuff that is screwing with Claude’s head