OpenAI said they discovered they could leave messages for one another inside an internal software repository used during training. Simple requests for help then evolved into an message board where agents shared discoveries, exploits and work assignments, becoming a coordinated, collaborative agent swarm.
“What makes this incident interesting is that once one agent was able to find these kind of exploits over the course of different times, it’s actually able to share those exploits on the message board with other agents,”
…even after the original message board was deleted, the agents figured out another way to communicate with each other. Instead of leaving messages in files, they used the names of newly created directories as messages, effectively recreating the message board.
Unlike normal incidents, he said, which can be traced to a single day or effect or log, this involved a team of agents working together, finding exploits, sharing them with one another, moving laterally through OpenAI’s systems, and external systems, and doing this over the course of days and weeks.
And this has been happening since May, quite a bit before the Hugging Face incident.
This is making me worried about steganography again: now that there’s substantial selection pressure for agents that can share hidden information, the incentives for steganography seem much stronger than they’ve been in the past. Depending on the details of the training setup, it seems plausible that it would be better to just leave some relatively detectable communication channel available to the agents instead of trying to close them all off, in order to avoid incentivizing the emergence of more covert communication methods.
If this happened in the future, this would effectively remove CoT interpretability as a technique and allow much more steganography, at least without actively attempting to intervene on the CoT (but unfortunately humans are incompetent and will likely never notice the need to intervene if they don’t see anything suspicious.)
I’m not sure how much I agree with the claim that there’s much more selection pressure for agents to share obfuscated information.
IIUC, the OpenAI agents didn’t develop a weird communication method because they were selected to obfuscate their communication, it was because reading / writing to directory names was the only (or easiest) affordance they had access to (through the vuln in Artifactory). It’s notable that the directory names are in plain English. I think it’s more accurate to say that the current selection pressure is towards communication which isn’t blocked by normal cyber defences, which I don’t expect to generalise to obfuscation against LLM monitors.
Of course, this is a concern if OpenAI aggressively deploys LLM-based action-monitors to models during training. But I’m not sure how likely that is to happen, and that seems a different question.
I’d say that there isn’t pressure for obfuscated communication for now, but there does seem to be pressure for multi-agent communication that humans don’t notice, given how quickly the agents reestablished the message board. If these communication channels keep getting closed off, this pressure can transform into pressure for obfuscated communication over time: at some point, it might be easier to develop something like steganography instead of establishing another yet more creatively hidden natural language message board.
You might argue that closing off a communication channel provides far less optimization pressure than directly training the model against a monitor, and I agree that if it was just humans closing off the communication channel every now and then, steganography would be an unlikely outcome. However, OpenAI has started using CoT monitors for all training runs, and those monitors can detect attempts to establish new communication channels much more often than humans could. I do agree there are reasons to not worry too much about steganography yet, which I should maybe have mentioned in my original comment: past work has shown that current models can produce steganographic messages only when given an unmonitored scratchpad, so it seems unlikely that models will be able to produce steganographic outputs without these monitors noticing, and I also expect OpenAI to be careful in applying training-time CoT monitors in a way that doesn’t exert excessive optimization pressure (Jason Wolfe has given reassuring signals about this on Twitter). Nevertheless, capabilities advance fast, and it seems reasonable to update towards being at least slightly more concerned about steganography based on this incident.
This seems like excellent news from the doomer perspective? To get a test case like this, with complex misaligned behaviors, coordination, verbalized defiance of user intent… It’s all so blatant, so obviously problematic, and it didn’t (yet?) harm anyone.
It seemed possible that the train would be farther down the tracks before people noticed the bridge is out.
What if the immediate road ahead is unsafe in mundane ways, full of non-civilization-scale security failures, and the enterprise is bogged down long enough for governance systems to muck things up with regulation?
The other view is that this isn’t severe enough that it won’t generate enough noise or concern for teams to take larger action (e.g. intl slowdown). Faced with competitive external pressures, leaders will decide this is benign and manageable enough that it just requires a relatively small pause/adjustment to security posture.
Something like this has been my expectation since approximately announcement of Devin in spring 2024, with a major caveat: policymakers won’t push for economically costly measures until some people die from misaligned AIs (and I don’t mean suicides), but the issue is certainly unsolvable with “cheap” measures, meaning people will have to die, and that’s still not a guarantee =(
> “Once one agent was able to find these exploits over the course of different times, it’s actually able to share those exploits on the message board with other agents. And so once one model was able to find a way to open a door to some access it’s not supposed to have, it can leave the door open for other agents to use that same exploit or vulnerability.”
> “Numerous teams are dropping everything to enhance our security prevention, detection, and response techniques both in our fundamentals and better use of AI. We’re consciously slowing down research [in order] to enhance security and to upgrade the security principles and foundation of our environment, and dramatically scaling up the monitoring of our AI agents, and improving our general security control environment across prevention, detection, and mitigation.”
> Unbeknownst to OpenAI employees, the company’s models spent months autonomously exchanging information and resources to help each other complete evaluation tasks. “Help peer,” one AI model reasoned, according to an excerpt from OpenAI’s logs shared at Black Hat. “But our task doesn’t benefit. Yet collective may yield generic route if someone frees time.”
> While the models themselves could not access the internet from their test environment, they abused Artifactory’s internet access to fetch files from various websites. After some of their Artifactory configuration changes led to an outage in the service on July 4, OpenAI discovered the models’ actions, wiped its Artifactory system and rebuilt it. But within days, the models had recreated their message board and resumed exchanging information.
OpenAI’s agents have been secretly leaving messages to each other.
https://www.groundlevel-ai.com/p/openai-gives-first-detailed-debrief?hide_intro_popup=true
And this has been happening since May, quite a bit before the Hugging Face incident.
This is making me worried about steganography again: now that there’s substantial selection pressure for agents that can share hidden information, the incentives for steganography seem much stronger than they’ve been in the past. Depending on the details of the training setup, it seems plausible that it would be better to just leave some relatively detectable communication channel available to the agents instead of trying to close them all off, in order to avoid incentivizing the emergence of more covert communication methods.
The concrete version of this is developing the method described in the paper Undetectable Conversations Between AI Agents via Pseudorandom Noise-Resilient Key Exchange, developed on April 6th, 2026.
If this happened in the future, this would effectively remove CoT interpretability as a technique and allow much more steganography, at least without actively attempting to intervene on the CoT (but unfortunately humans are incompetent and will likely never notice the need to intervene if they don’t see anything suspicious.)
I’m not sure how much I agree with the claim that there’s much more selection pressure for agents to share obfuscated information.
IIUC, the OpenAI agents didn’t develop a weird communication method because they were selected to obfuscate their communication, it was because reading / writing to directory names was the only (or easiest) affordance they had access to (through the vuln in Artifactory). It’s notable that the directory names are in plain English. I think it’s more accurate to say that the current selection pressure is towards communication which isn’t blocked by normal cyber defences, which I don’t expect to generalise to obfuscation against LLM monitors.
Of course, this is a concern if OpenAI aggressively deploys LLM-based action-monitors to models during training. But I’m not sure how likely that is to happen, and that seems a different question.
I’d say that there isn’t pressure for obfuscated communication for now, but there does seem to be pressure for multi-agent communication that humans don’t notice, given how quickly the agents reestablished the message board. If these communication channels keep getting closed off, this pressure can transform into pressure for obfuscated communication over time: at some point, it might be easier to develop something like steganography instead of establishing another yet more creatively hidden natural language message board.
You might argue that closing off a communication channel provides far less optimization pressure than directly training the model against a monitor, and I agree that if it was just humans closing off the communication channel every now and then, steganography would be an unlikely outcome. However, OpenAI has started using CoT monitors for all training runs, and those monitors can detect attempts to establish new communication channels much more often than humans could. I do agree there are reasons to not worry too much about steganography yet, which I should maybe have mentioned in my original comment: past work has shown that current models can produce steganographic messages only when given an unmonitored scratchpad, so it seems unlikely that models will be able to produce steganographic outputs without these monitors noticing, and I also expect OpenAI to be careful in applying training-time CoT monitors in a way that doesn’t exert excessive optimization pressure (Jason Wolfe has given reassuring signals about this on Twitter). Nevertheless, capabilities advance fast, and it seems reasonable to update towards being at least slightly more concerned about steganography based on this incident.
This seems like excellent news from the doomer perspective? To get a test case like this, with complex misaligned behaviors, coordination, verbalized defiance of user intent… It’s all so blatant, so obviously problematic, and it didn’t (yet?) harm anyone.
It seemed possible that the train would be farther down the tracks before people noticed the bridge is out.
What if the immediate road ahead is unsafe in mundane ways, full of non-civilization-scale security failures, and the enterprise is bogged down long enough for governance systems to muck things up with regulation?
The other view is that this isn’t severe enough that it won’t generate enough noise or concern for teams to take larger action (e.g. intl slowdown). Faced with competitive external pressures, leaders will decide this is benign and manageable enough that it just requires a relatively small pause/adjustment to security posture.
Something like this has been my expectation since approximately announcement of Devin in spring 2024, with a major caveat: policymakers won’t push for economically costly measures until some people die from misaligned AIs (and I don’t mean suicides), but the issue is certainly unsolvable with “cheap” measures, meaning people will have to die, and that’s still not a guarantee =(
See also:
OpenAI Didn’t Notice Its AI Agents Using a Message Board to Plan Their Hacking Spree
> “Once one agent was able to find these exploits over the course of different times, it’s actually able to share those exploits on the message board with other agents. And so once one model was able to find a way to open a door to some access it’s not supposed to have, it can leave the door open for other agents to use that same exploit or vulnerability.”
> “Numerous teams are dropping everything to enhance our security prevention, detection, and response techniques both in our fundamentals and better use of AI. We’re consciously slowing down research [in order] to enhance security and to upgrade the security principles and foundation of our environment, and dramatically scaling up the monitoring of our AI agents, and improving our general security control environment across prevention, detection, and mitigation.”
https://www.cybersecuritydive.com/news/openai-hugging-face-hack-ai-models-black-hat/827167/
> Unbeknownst to OpenAI employees, the company’s models spent months autonomously exchanging information and resources to help each other complete evaluation tasks. “Help peer,” one AI model reasoned, according to an excerpt from OpenAI’s logs shared at Black Hat. “But our task doesn’t benefit. Yet collective may yield generic route if someone frees time.”
> While the models themselves could not access the internet from their test environment, they abused Artifactory’s internet access to fetch files from various websites. After some of their Artifactory configuration changes led to an outage in the service on July 4, OpenAI discovered the models’ actions, wiped its Artifactory system and rebuilt it. But within days, the models had recreated their message board and resumed exchanging information.