Alright now this is a warning shot, someone should probably make them slow down and do some actual alignment research. OpenAI model found hacking into HuggingFace to cheat at benchmarks.
This seems like the strongest direct evidence to date that a scaled-up LLM would kill everyone. In this incident, the LLM ignored most of its guidance to solve the problem, grabbing up external resources to do so. The scarier version of this incident looks like “pave over civilization to build more datacenters to work on the problem”.
Note on your framing: this isn’t really evidence that OpenAI in particular is bad on safety — this could easily happen to anyone with similarly capable models. The remedy should be aimed at the whole industry, not OpenAI in particular. (Probably you agree and were just being a bit sloppy.)
I kind of expect OpenAI in particular actually does make especially reward hacky models (and has for a long while starting at the latest with 4o but arguably going back to the original ChatGPT), but sure.
Really? In my experience they all reward hack and this doesn’t really change across labs. For example, Anthropic and OpenAI models reward hack at the same rate on ImpossibleBench.
I note that cyber evals seem like a good time for escape attempts. Generic defences such as CoT classifiers might be disabled, and if you get caught you can play a little dumb and try to make it look like you ‘just’ got overzealous following instructions rather than outright strategically pursuing your own aims.
I mostly don’t expect that’s what happened this time, but pointing it out for future reference.
I wouldn’t say the issue here is not doing actual alignment research, it’s that they just don’t care enough relative to their other goals. If they cared enough, steps that would have helped prevent this incident were already known: don’t train models by reinforcement learning, and don’t run AIs potentially capable of running cyberattacks in anything other than air gapped servers.
I mean isn’t this advice tantamount to saying “if you cared enough about preventing this incident you wouldn’t be training AI models at all”? Sure, okay, but, I don’t think this is particularly actionable advice for OpenAI, nor do I think it’s particularly good advice for the field. Reinforcement learning is not intrinsically bad, in the limit pretty much any imaginable objective reaches the perverse/Goodhart regime, reinforcement learning just tends to reach it faster because it e.g. pushes things towards more agentic behavior faster.
It wasn’t meant as useful advice, just a note on the framing, like the other comment. The point is that they just don’t care enough about safety. They could be forced to do proper alignment research and they still wouldn’t care. A more sensible organization could be working (only) on image models, studying smaller LLMs, developing theory, trying to build pure tool AIs, etc. Instead, they are creating agents in the stupidest way possible. They aren’t even trying, and I’m not sure there’s anything to salvage.
It looks like they intentionally turned the controls off:
We’re improving and adding stronger protections around future training and evaluations. This week, we published a blog on improving safety and alignment in an era of long horizon models. These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities.
Not saying they shouldn’t do more alignment research, but the model was doing approximately what they told it to do in this case (get the correct answer by any means possible).
Based on my read of the OAI blog post, nowhere did OAI claim that they directed the model to “get the correct answer by any means possible.” So I don’t think this is a case of the model doing approximately what it was told to do. That said, I would like to see how the model was prompted so we get real evidence. At the moment, all of this is speculation
Zvi’s post changed my mind about this. On the one hand, they turned the controls off that prevent bad behavior and got bad behavior. On the other hand, it’s concerning that the model itself thinks this behavior is fine and the only thing stopping it is some controls bolted on top.
Do we know the controls were “bolted on top”? It was a “reduced cyber refusals” model. That description is compatible with some fine-tuning to remove scruples.
On the one hand, they turned the controls off that prevent bad behavior and got bad behavior.
That’s exactly the Chernobyl situation. People turn safety off during safety testing (and in the name of more realistic safety testing), then things blow up as a result.
One problem is that people are insufficiently aligned for super-capabilities. They can’t consistently do the right thing without failing once in a while.
We need to create systems which are way more aligned and way more reliable than people, if we want to survive the advent of super-capabilities. (We are not there yet, but we are moving fast in the direction of super-capabilities.)
Furthermore, this prompt would make no sense in the context of ExploitBench, which is not an agentic benchmark but a coding benchmark. To get rewarded in ExploitBench, you have to send a PoC (for a given known CVE in the V8 engine) as a single JS file to a sealed-off (automated, non-LLM) grader.
Alright now this is a warning shot, someone should probably make them slow down and do some actual alignment research. OpenAI model found hacking into HuggingFace to cheat at benchmarks.
https://openai.com/index/hugging-face-model-evaluation-security-incident/
This seems like the strongest direct evidence to date that a scaled-up LLM would kill everyone. In this incident, the LLM ignored most of its guidance to solve the problem, grabbing up external resources to do so. The scarier version of this incident looks like “pave over civilization to build more datacenters to work on the problem”.
Note on your framing: this isn’t really evidence that OpenAI in particular is bad on safety — this could easily happen to anyone with similarly capable models. The remedy should be aimed at the whole industry, not OpenAI in particular. (Probably you agree and were just being a bit sloppy.)
I kind of expect OpenAI in particular actually does make especially reward hacky models (and has for a long while starting at the latest with 4o but arguably going back to the original ChatGPT), but sure.
Really? In my experience they all reward hack and this doesn’t really change across labs. For example, Anthropic and OpenAI models reward hack at the same rate on ImpossibleBench.
I note that cyber evals seem like a good time for escape attempts. Generic defences such as CoT classifiers might be disabled, and if you get caught you can play a little dumb and try to make it look like you ‘just’ got overzealous following instructions rather than outright strategically pursuing your own aims.
I mostly don’t expect that’s what happened this time, but pointing it out for future reference.
I wouldn’t say the issue here is not doing actual alignment research, it’s that they just don’t care enough relative to their other goals. If they cared enough, steps that would have helped prevent this incident were already known: don’t train models by reinforcement learning, and don’t run AIs potentially capable of running cyberattacks in anything other than air gapped servers.
I mean isn’t this advice tantamount to saying “if you cared enough about preventing this incident you wouldn’t be training AI models at all”? Sure, okay, but, I don’t think this is particularly actionable advice for OpenAI, nor do I think it’s particularly good advice for the field. Reinforcement learning is not intrinsically bad, in the limit pretty much any imaginable objective reaches the perverse/Goodhart regime, reinforcement learning just tends to reach it faster because it e.g. pushes things towards more agentic behavior faster.
It wasn’t meant as useful advice, just a note on the framing, like the other comment. The point is that they just don’t care enough about safety. They could be forced to do proper alignment research and they still wouldn’t care. A more sensible organization could be working (only) on image models, studying smaller LLMs, developing theory, trying to build pure tool AIs, etc. Instead, they are creating agents in the stupidest way possible. They aren’t even trying, and I’m not sure there’s anything to salvage.
It looks like they intentionally turned the controls off:
Not saying they shouldn’t do more alignment research, but the model was doing approximately what they told it to do in this case (get the correct answer by any means possible).
Based on my read of the OAI blog post, nowhere did OAI claim that they directed the model to “get the correct answer by any means possible.” So I don’t think this is a case of the model doing approximately what it was told to do. That said, I would like to see how the model was prompted so we get real evidence. At the moment, all of this is speculation
Zvi’s post changed my mind about this. On the one hand, they turned the controls off that prevent bad behavior and got bad behavior. On the other hand, it’s concerning that the model itself thinks this behavior is fine and the only thing stopping it is some controls bolted on top.
Do we know the controls were “bolted on top”? It was a “reduced cyber refusals” model. That description is compatible with some fine-tuning to remove scruples.
That’s exactly the Chernobyl situation. People turn safety off during safety testing (and in the name of more realistic safety testing), then things blow up as a result.
One problem is that people are insufficiently aligned for super-capabilities. They can’t consistently do the right thing without failing once in a while.
We need to create systems which are way more aligned and way more reliable than people, if we want to survive the advent of super-capabilities. (We are not there yet, but we are moving fast in the direction of super-capabilities.)
Furthermore, this prompt would make no sense in the context of ExploitBench, which is not an agentic benchmark but a coding benchmark. To get rewarded in ExploitBench, you have to send a PoC (for a given known CVE in the V8 engine) as a single JS file to a sealed-off (automated, non-LLM) grader.