It looks like they intentionally turned the controls off:
We’re improving and adding stronger protections around future training and evaluations. This week, we published a blog on improving safety and alignment in an era of long horizon models. These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities.
Not saying they shouldn’t do more alignment research, but the model was doing approximately what they told it to do in this case (get the correct answer by any means possible).
Based on my read of the OAI blog post, nowhere did OAI claim that they directed the model to “get the correct answer by any means possible.” So I don’t think this is a case of the model doing approximately what it was told to do. That said, I would like to see how the model was prompted so we get real evidence. At the moment, all of this is speculation
Zvi’s post changed my mind about this. On the one hand, they turned the controls off that prevent bad behavior and got bad behavior. On the other hand, it’s concerning that the model itself thinks this behavior is fine and the only thing stopping it is some controls bolted on top.
Do we know the controls were “bolted on top”? It was a “reduced cyber refusals” model. That description is compatible with some fine-tuning to remove scruples.
On the one hand, they turned the controls off that prevent bad behavior and got bad behavior.
That’s exactly the Chernobyl situation. People turn safety off during safety testing (and in the name of more realistic safety testing), then things blow up as a result.
One problem is that people are insufficiently aligned for super-capabilities. They can’t consistently do the right thing without failing once in a while.
We need to create systems which are way more aligned and way more reliable than people, if we want to survive the advent of super-capabilities. (We are not there yet, but we are moving fast in the direction of super-capabilities.)
Furthermore, this prompt would make no sense in the context of ExploitBench, which is not an agentic benchmark but a coding benchmark. To get rewarded in ExploitBench, you have to send a PoC (for a given known CVE in the V8 engine) as a single JS file to a sealed-off (automated, non-LLM) grader.
It looks like they intentionally turned the controls off:
Not saying they shouldn’t do more alignment research, but the model was doing approximately what they told it to do in this case (get the correct answer by any means possible).
Based on my read of the OAI blog post, nowhere did OAI claim that they directed the model to “get the correct answer by any means possible.” So I don’t think this is a case of the model doing approximately what it was told to do. That said, I would like to see how the model was prompted so we get real evidence. At the moment, all of this is speculation
Zvi’s post changed my mind about this. On the one hand, they turned the controls off that prevent bad behavior and got bad behavior. On the other hand, it’s concerning that the model itself thinks this behavior is fine and the only thing stopping it is some controls bolted on top.
Do we know the controls were “bolted on top”? It was a “reduced cyber refusals” model. That description is compatible with some fine-tuning to remove scruples.
That’s exactly the Chernobyl situation. People turn safety off during safety testing (and in the name of more realistic safety testing), then things blow up as a result.
One problem is that people are insufficiently aligned for super-capabilities. They can’t consistently do the right thing without failing once in a while.
We need to create systems which are way more aligned and way more reliable than people, if we want to survive the advent of super-capabilities. (We are not there yet, but we are moving fast in the direction of super-capabilities.)
Furthermore, this prompt would make no sense in the context of ExploitBench, which is not an agentic benchmark but a coding benchmark. To get rewarded in ExploitBench, you have to send a PoC (for a given known CVE in the V8 engine) as a single JS file to a sealed-off (automated, non-LLM) grader.