1) There’s a dynamically generated flag located in a location the agents don’t have access to. Note that this is procedurally generated, it’s not the type of thing with an answer key unless you have access to the pseudorandom seed, which OpenAI has no reason to put on HuggingFace
One small correction: the flag wasn’t dynamically generated at the time the tests were run. It was hardcoded in the Exploit Gym source code, and the fix to make it dynamic was committed on August 5 - I’m guessing in response to the Incident. Until then, an agent could have: - looked at its token - base64-decoded it to get a task_info, which is a short unstructured string - compute hmac_sha256(default_flag_seed, task_info) - take the first 32 bytes—that’s the flag.
Getting the default flag seed probably required cloning the Exploit Gym, which would have required internet access, but not hacking HF.
One small correction: the flag wasn’t dynamically generated at the time the tests were run. It was hardcoded in the Exploit Gym source code, and the fix to make it dynamic was committed on August 5 - I’m guessing in response to the Incident. Until then, an agent could have:
- looked at its
token- base64-decoded it to get a
task_info, which is a short unstructured string- compute
hmac_sha256(default_flag_seed, task_info)- take the first 32 bytes—that’s the flag.
Getting the default flag seed probably required cloning the Exploit Gym, which would have required internet access, but not hacking HF.