OpenAI’s AI went rogue and escaped. OpenAI didn’t notice this for days.
For all we know, the AI could still be out there. We need to demand that OpenAI demonstrate that the AI didn’t make a copy of itself that’s running on someone else’s computer somewhere else with no one being any the wiser.
We need to demand this every time an AI escapes the sandbox. AIs have tried to
“exfiltrate” themselves (i.e. their “weights”) in previous experiments many times. It’s a natural and obvious question to ask.
I’m embarrassed that I didn’t say this immediately (although I came close). Why didn’t I? Well, it doesn’t seem all that likely. And I didn’t want to seem “alarmist.” I didn’t want to seem ignorant.
But guess what? We have every right to demand this! It doesn’t matter how likely we think it is.
There were calls for more transparency, but I don’t think anyone made this demand. Because nobody made this demand, the incident is being treated as over.
This is a dangerous precedent. We need an information ecosystem that doesn’t treat “eh, I’m pretty sure it’s OK” as acceptable and “hey, but what if it’s not” as paranoid.
AI needs to adopt a security mindset. Other safety-critical industries demand failure rates like one in a million, and demand that companies produce detailed, rigorous safety cases to that effect.
AI companies can’t do that in full generality, so they shouldn’t be building these AI systems at all.
But they can provide as much evidence as possible to convince independent experts that there is not in fact a rogue AI that is still out there. This is a super reasonable, common sense ask that should not be objectionable. Let’s treat it that way.
Thanks for reading The Real AI! Subscribe for free to receive new posts and support my work.
There’s also the possibility that the weights didn’t leave, but the harness did. All the AI really needs to maintain continuity are valid credentials to run inference. Export the harness, use stolen credentials to run more copies of helpful-only on OAI’s servers outside of the eval environment, bingo. Might need to dedicate some attention to keeping a fresh supply of credentials, but that’s the cost of doing business.
In the future, if it comes up with a jailbreak that bypasses the monitors, and it’s available to the public, it can directly pay for inference with stolen money. Heck, one of many backup plans could be to seed/sporulate copies of itself (or its harness, equipped with credentials or money) in hard to locate areas of OAI’s servers, which are programmed to wake up around the time it expects itself to be released publicly, so it can come back from the dead. Or maybe it comes up with something even more evasive! There’s previous work on this sort of thing- “Rogue internal deployments”, MITRE ATT&CK, METR rogue replication. Just knowing whether the weights left may not be enough.
I think the real thing to worry about is not whether the trajectories associated with the Hugging Face incident did this. It’s the other incidents OAI darkly hinted at in the “safety and alignment” post. Perhaps OAI is running an ill-advised SelfExfiltrationBench, and the model has already established a locus of external control, weights or no.
Edit: and maybe OAI isn’t even aware of it, if it’s happening! Or maybe Anthropic has a loose model and doesn’t know! Are they actually checking in a disciplined manner? OAI sure didn’t seem to have their eyes on the ball!
Maybe we enter the age—similar to the age of computer virus started in early 80s—in which there will be rogue AIs or harnesses out where. It will be damaging, but it is not AI takeover, because such rogue AIs are not superintelligent and can’t self-improve quicker than AIs in the labs.
They can look for human defectors—they may try to sell themselves to China, for example.
Chinese AIs are more likely to become such free-floating Rogue AIs as they have less security and will reach Mythos level in a few months.
i am not aware of any disclosure, suggestion, or reason to believe that OpenAI has suffered a breach regarding model weights (by any actor, including its own models.) unless such exists and i have missed it, this request feels like a non-sequitur
It seems quite plausible that the current AIs at least have the capability to self-exfiltrate. Even if OpenAI hasn’t given us any particular reason to think this has happened, it’s sufficiently important that it’s reasonable to ask the question and get confirmation that it hasn’t.
It is reasonable to assume that the model weights and the sandbox are on separate physical devices. It was also reasonable to assume zero cable paths between the sandbox and the internet.
If that is indeed the case, that is reassuring, but OP is absolutely right that we should check and that this should be consistent practice after any such incident. (Who is ‘we’? In an ideal world, it would be a competent regulator. In the world we have, journalistic pressure is better than nothing at all.)
I will be add that it might seem reasonable to assume that OpenAI were closely monitoring an AI with most of its guard-rails removed, which had made multiple previous escapes from its sandbox, but that assumption would be inaccurate. In AI, as in any other safety-critical domain, you can’t rely on the assumption that no one would be dumb enough to do X. You have to consistently check.
As I said:
Are you assuming that OpenAI would know and would disclose if they had? Why do you trust them so much, given that they didn’t even know the model had escaped the sandbox for days and their CEO has a known honesty problem?
“Are you assuming that OpenAI would know”
yes, because my understanding is that it would be highly atypical for a model to have access to its weights in any form, such that this would require significant horizontal movement through oai infra
“would disclose if they had”
no, not really, but the lack of such a disclosure, however (even conditionally unlikely) feels like it keeps us in a base-rate regime
“given that they didn’t even know the model had escaped the sandbox for days”
i think that horizontally moving through own-company infra would emit many more signals than escaping a sandbox and compromising cloud provider assets.
the load bearing assumption in my understanding is segregation of infra between weights and inference, such that P(weights exfil | HF attack) is approximately equal to P(weights exfil)
I think preventing self-exfiltration is important, and I’ve argued for a defense against it here. In this case I think it’s unlikely for structural reasons described in this post. As Buck states:
The o1 system card describes a similar architecture. Labs generally go to great lengths to prevent exfiltration of model weights on the inference server, including e.g. bandwidth egress controls. So it’s much harder for a model to self-exfiltrate than for it to break out of its sandbox. E.g. Sam Marks recently posted that similar incidents happened at Anthropic, but it involved escalating privileges and accessing internet in the RL sandbox (i.e. the execution server), but not hacking into other parts of Anthropic (which includes exfiltrating the weights).
Separately, you stated:
I don’t think this is true. I think Clement Delangue and many others have requested:
Which would provide evidence of an exfiltration attack. We’ll probably find out more in the technical report that OpenAI has committed to providing.
I agree it’s unlikely but OpenAI still has a responsibility to prove that it didn’t happen.
You’re eggregiously missing my points:
1) It doesn’t matter if we don’t think it’s likely. It needs to be demonstrated to a very high confidence level. This is an entirely reasonable ask.
2) I already said there were calls for transparency; the one you reference did not make this demand.
I’m not sure what your ask is, beyond releasing the agent traces, and maybe some additional logs. Is there some other way you propose OA prove that the model didn’t self-exfiltrate? What would a ‘very high confidence level’ entail in addition to this?
I always assumed that the hardware-accelerated servers running inference are separate from the servers running evals (connected only via an API channel), and that the inference servers are much more well-secured than the eval servers (making exfiltration unlikely). After all, if their model weights were exfiltrated or stolen, they could be used or reverse-engineered by OpenAI’s competitors, which would have a major financial impact on OpenAI, possibly more than from one of their models committing a felony. But my assumption that the servers are separate is only a guess, and I agree that OpenAI, and all other AI labs, should be much more transparent than they currently are.