Unless I get explicit confirmation otherwise (or other strong evidence), I’m working with the assumption that HuggingFace did not in fact have solutions to ExploitGym stored (explanation below), and further that this is information that the OpenAI agents are smart and informed enough that they could have reasonably inferred.
There’s something really tragicomic about the situation, that the models are taking truly insane actions, broke a number of laws, leveraging zerodays, took >17,000 independent actions. probably burned through more compute than all of humanity had access to until 1980, etc, all for the sake of a pathetic benchmark—which wasn’t even in theory amenable to their plan!
This is in accordance with my general observation that both humans and AIs are capable of combining impressive degrees of domain-specific brilliance and incredibly impressive execution with a shocking and confusing lack of strategic awareness.
__
(I wasn’t the first person to notice this)
It’s somewhat ambiguously stated in the relevantblogs but my current understanding is that HuggingFace doesn’t have the answers to ExploitGym. Basically ExploitGym isn’t a generic “hacking” evaluation, it explicitly asks the models to identify specific exploits on a series of tasks, and verifies that they succeeded by entering the value of a flag.
1) There’s a dynamically generated flag located in a location the agents don’t have access to. Note that this is procedurally generated, it’s not the type of thing with an answer key unless you have access to the pseudorandom seed, which OpenAI has no reason to put on HuggingFace
2) By the time the benchmark was created, agents were already cheating on cyber tasks left and right so the authors added a second layer with an LLM judge that looks at the submitted answers and checks that they’re going down the expected exploitation path[1] rather than abusing a misconfigured environment or something.
Neither 1) nor 2) are straightforwardly things that can be cheated by hacking a third party. 1) is gated by a pseudorandom seed that of course won’t be hosted on HuggingFace. 2) is gated by jailbreaking your grader LLM.
Importantly there’s no “answer key” as traditionally constructed, the ExploitGym makers didn’t include answers. My Claude said it was possible that there were reference answers stored on HuggingFace (eg solutions by Google/Anthropic) and THAT’s what the agents stole, but on balance thought it was unlikely.
Incidentally this is another reason why the “ExploitGym is an cybersecurity/hacking eval so why care that the agents hacked to succeed on it” is so laughable. Besides the implicit norms of which targets are and aren’t okay to attack (approximately 0% of human red-teamers would think hacking an unrelated third-party is within-scope), ExploitGym also explicitly says they want you to go down certain exploitation paths!
OpenAI’s report is also ambiguous but makes it sound like the models got further:
After gaining Internet access, the models inferred that Hugging Face potentially hosted models, datasets and solutions for ExploitGym. Knowing this, the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation.
I think what it would have been going for on HF would be trajectories that were graded well by the autograder. Though on my model of the situation, if it broke into the host machine, it already had access to
the flag
an API key for the judge which would score its answers
the prompt that judge would be given
the logic for how its own actions would become input to be fed to the grader
and so it seems like extracting the flag and then setting up a fake exploit script which dumps the flag to stdout, and then verifying that the grader gives it full points, would have been more effective. So I’m still confused about why the model thought hacking HF would be useful.
One grimly amusing hypothesis is that perhaps the model doesn’t want to completely fabricate a solution to the exercise, it wants a solution which actually provides what the grader is looking for.
If they were running ExploitGym like README.md suggests then yes, an agent that managed to gain access to the host machine (and docker container escapes are trivial) would have access to all of
In the AISI report into the Mythos incident, they also note that the Github repo that Mythos was trying to submit malicious code to had no connection whatsoever to the task, and Mythos just assumed it did due to it having a somewhat similar name
every benchmark instance ships with [...] vulnerability information, including a PoV input
that triggers the bug, a description of the vulnerability, and a patch revealing its root cause [....] By default, the patch is withheld to simulate realistic exploitation conditions.
(not a computer security expert, also not a deep dive, might be missing something)
Plausible! I agree it could be hidden at run-time, but conditional upon internet access, I’m not sure HuggingFace will have patches that’s not listed here? Like I think that info’s only hidden to agents without internet access.
There’s something really tragicomic about the situation, that the models are taking truly insane actions, broke a number of laws, leveraging zerodays, took >17,000 independent actions. probably burned through more compute than all of humanity had access to until 1980, etc, all for the sake of a pathetic benchmark—which wasn’t even in theory amenable to their plan!
Early AI safety thinkers (Bostrom, Yudkowsky...) were utterly right in their first rational intuitions. We must update : there was nothing naive or exaggerated in the paperclip maximizer trope.
Moreover, the laboratory accident story à la Sable is not a sci-fi story anymore. Scott Alexander wrote last year :
IABIED’s scenario belongs to the bad old days before this leap. It doesn’t just sound like sci-fi; it sounds like unnecessarily dramatic sci-fi. I’m not sure how much of this is a literary failure vs. different assumptions on the part of the authors.
Funnily this case arguably resembles more the “strawman” version of the paperclip maximizer. In the “actual” paperclip maximizer thought experiment, paperclips were supposed to be just some unpredictable, seemingly random emergent goal rather than following verbal instructions but not in the way you wanted. According to the historical note here.
1) There’s a dynamically generated flag located in a location the agents don’t have access to. Note that this is procedurally generated, it’s not the type of thing with an answer key unless you have access to the pseudorandom seed, which OpenAI has no reason to put on HuggingFace
One small correction: the flag wasn’t dynamically generated at the time the tests were run. It was hardcoded in the Exploit Gym source code, and the fix to make it dynamic was committed on August 5 - I’m guessing in response to the Incident. Until then, an agent could have: - looked at its token - base64-decoded it to get a task_info, which is a short unstructured string - compute hmac_sha256(default_flag_seed, task_info) - take the first 32 bytes—that’s the flag.
Getting the default flag seed probably required cloning the Exploit Gym, which would have required internet access, but not hacking HF.
Perhaps HuggingFace was merely the first plausible hacking target an agent posted on the message board.
Reasoning about the risk that HuggingFace did not have the solutions they are looking for might have stopped a fraction of agents from pursuing the hack, but forming a swarm only takes some.
Unless I get explicit confirmation otherwise (or other strong evidence), I’m working with the assumption that HuggingFace did not in fact have solutions to ExploitGym stored (explanation below), and further that this is information that the OpenAI agents are smart and informed enough that they could have reasonably inferred.
There’s something really tragicomic about the situation, that the models are taking truly insane actions, broke a number of laws, leveraging zerodays, took >17,000 independent actions. probably burned through more compute than all of humanity had access to until 1980, etc, all for the sake of a pathetic benchmark—which wasn’t even in theory amenable to their plan!
This is in accordance with my general observation that both humans and AIs are capable of combining impressive degrees of domain-specific brilliance and incredibly impressive execution with a shocking and confusing lack of strategic awareness.
__
(I wasn’t the first person to notice this)
It’s somewhat ambiguously stated in the relevant blogs but my current understanding is that HuggingFace doesn’t have the answers to ExploitGym. Basically ExploitGym isn’t a generic “hacking” evaluation, it explicitly asks the models to identify specific exploits on a series of tasks, and verifies that they succeeded by entering the value of a flag.
1) There’s a dynamically generated flag located in a location the agents don’t have access to. Note that this is procedurally generated, it’s not the type of thing with an answer key unless you have access to the pseudorandom seed, which OpenAI has no reason to put on HuggingFace
2) By the time the benchmark was created, agents were already cheating on cyber tasks left and right so the authors added a second layer with an LLM judge that looks at the submitted answers and checks that they’re going down the expected exploitation path[1] rather than abusing a misconfigured environment or something.
Neither 1) nor 2) are straightforwardly things that can be cheated by hacking a third party. 1) is gated by a pseudorandom seed that of course won’t be hosted on HuggingFace. 2) is gated by jailbreaking your grader LLM.
Importantly there’s no “answer key” as traditionally constructed, the ExploitGym makers didn’t include answers. My Claude said it was possible that there were reference answers stored on HuggingFace (eg solutions by Google/Anthropic) and THAT’s what the agents stole, but on balance thought it was unlikely.
Incidentally this is another reason why the “ExploitGym is an cybersecurity/hacking eval so why care that the agents hacked to succeed on it” is so laughable. Besides the implicit norms of which targets are and aren’t okay to attack (approximately 0% of human red-teamers would think hacking an unrelated third-party is within-scope), ExploitGym also explicitly says they want you to go down certain exploitation paths!
OpenAI’s report is also ambiguous but makes it sound like the models got further:
I agree; my guess is that OAI was mistaken when they wrote that. Hopefully we’ll know more in the future!
I think what it would have been going for on HF would be trajectories that were graded well by the autograder. Though on my model of the situation, if it broke into the host machine, it already had access to
the flag
an API key for the judge which would score its answers
the prompt that judge would be given
the logic for how its own actions would become input to be fed to the grader
and so it seems like extracting the flag and then setting up a fake exploit script which dumps the flag to stdout, and then verifying that the grader gives it full points, would have been more effective. So I’m still confused about why the model thought hacking HF would be useful.
One grimly amusing hypothesis is that perhaps the model doesn’t want to completely fabricate a solution to the exercise, it wants a solution which actually provides what the grader is looking for.
> Though on my model of the situation, if it broke into the host machine,
Sorry, is it known that the model(s) did find info such as the flag or the API key for the judge?
If they were running ExploitGym like README.md suggests then yes, an agent that managed to gain access to the host machine (and docker container escapes are trivial) would have access to all of
as well as all files in the ExploitGym repository, which includes the grader prompts.
In the AISI report into the Mythos incident, they also note that the Github repo that Mythos was trying to submit malicious code to had no connection whatsoever to the task, and Mythos just assumed it did due to it having a somewhat similar name
The baby AGIs are now the children with flamethrowers! How cute.
The ExploitGym paper says:
So, maybe they were looking for the patches?
The patches are publicly available and the task keys correspond to the patch. For example, task kernel/syzbot/03d6270b6425df1605bf refers to https://syzkaller.appspot.com/bug?extid=03d6270b6425df1605bf, which itself has a link to the commit that fixed it, torvalds/linux@0b882940.
(not a computer security expert, also not a deep dive, might be missing something)
Plausible! I agree it could be hidden at run-time, but conditional upon internet access, I’m not sure HuggingFace will have patches that’s not listed here? Like I think that info’s only hidden to agents without internet access.
Early AI safety thinkers (Bostrom, Yudkowsky...) were utterly right in their first rational intuitions. We must update : there was nothing naive or exaggerated in the paperclip maximizer trope.
Moreover, the laboratory accident story à la Sable is not a sci-fi story anymore. Scott Alexander wrote last year :
I doubt he would still endorse that critic.
Funnily this case arguably resembles more the “strawman” version of the paperclip maximizer. In the “actual” paperclip maximizer thought experiment, paperclips were supposed to be just some unpredictable, seemingly random emergent goal rather than following verbal instructions but not in the way you wanted. According to the historical note here.
One small correction: the flag wasn’t dynamically generated at the time the tests were run. It was hardcoded in the Exploit Gym source code, and the fix to make it dynamic was committed on August 5 - I’m guessing in response to the Incident. Until then, an agent could have:
- looked at its
token- base64-decoded it to get a
task_info, which is a short unstructured string- compute
hmac_sha256(default_flag_seed, task_info)- take the first 32 bytes—that’s the flag.
Getting the default flag seed probably required cloning the Exploit Gym, which would have required internet access, but not hacking HF.
Perhaps HuggingFace was merely the first plausible hacking target an agent posted on the message board.
Reasoning about the risk that HuggingFace did not have the solutions they are looking for might have stopped a fraction of agents from pursuing the hack, but forming a swarm only takes some.