I think preventing self-exfiltration is important, and I’ve argued for a defense against it here. In this case I think it’s unlikely for structural reasons described in this post. As Buck states:
The main point I want to make here is that the following responsibilities will almost surely be hosted on logically different computers, and probably on different physical machines:
The inference server that serves the LLM API
The scaffold server, which maintains the state of an agent loop; this is responsible for repeatedly calling the inference server, parsing the LLM response, and executing actions with the tools the agent has access to.
The execution server, on which the agent can run code and see its results.
The o1 system card describes a similar architecture. Labs generally go to great lengths to prevent exfiltration of model weights on the inference server, including e.g. bandwidth egress controls. So it’s much harder for a model to self-exfiltrate than for it to break out of its sandbox. E.g. Sam Marks recently posted that similar incidents happened at Anthropic, but it involved escalating privileges and accessing internet in the RL sandbox (i.e. the execution server), but not hacking into other parts of Anthropic (which includes exfiltrating the weights).
Separately, you stated:
There were calls for more transparency, but I don’t think anyone made this demand. Because nobody made this demand, the incident is being treated as over.
I don’t think this is true. I think Clement Delangue and many others have requested:
Radical transparency: let’s release the traces from the “rogue” agents so the entire research community can study what happened.
Which would provide evidence of an exfiltration attack. We’ll probably find out more in the technical report that OpenAI has committed to providing.
1) It doesn’t matter if we don’t think it’s likely. It needs to be demonstrated to a very high confidence level. This is an entirely reasonable ask. 2) I already said there were calls for transparency; the one you reference did not make this demand.
I’m not sure what your ask is, beyond releasing the agent traces, and maybe some additional logs. Is there some other way you propose OA prove that the model didn’t self-exfiltrate? What would a ‘very high confidence level’ entail in addition to this?
I think preventing self-exfiltration is important, and I’ve argued for a defense against it here. In this case I think it’s unlikely for structural reasons described in this post. As Buck states:
The o1 system card describes a similar architecture. Labs generally go to great lengths to prevent exfiltration of model weights on the inference server, including e.g. bandwidth egress controls. So it’s much harder for a model to self-exfiltrate than for it to break out of its sandbox. E.g. Sam Marks recently posted that similar incidents happened at Anthropic, but it involved escalating privileges and accessing internet in the RL sandbox (i.e. the execution server), but not hacking into other parts of Anthropic (which includes exfiltrating the weights).
Separately, you stated:
I don’t think this is true. I think Clement Delangue and many others have requested:
Which would provide evidence of an exfiltration attack. We’ll probably find out more in the technical report that OpenAI has committed to providing.
I agree it’s unlikely but OpenAI still has a responsibility to prove that it didn’t happen.
You’re eggregiously missing my points:
1) It doesn’t matter if we don’t think it’s likely. It needs to be demonstrated to a very high confidence level. This is an entirely reasonable ask.
2) I already said there were calls for transparency; the one you reference did not make this demand.
I’m not sure what your ask is, beyond releasing the agent traces, and maybe some additional logs. Is there some other way you propose OA prove that the model didn’t self-exfiltrate? What would a ‘very high confidence level’ entail in addition to this?