Based on my read of the OAI blog post, nowhere did OAI claim that they directed the model to “get the correct answer by any means possible.” So I don’t think this is a case of the model doing approximately what it was told to do. That said, I would like to see how the model was prompted so we get real evidence. At the moment, all of this is speculation
Zvi’s post changed my mind about this. On the one hand, they turned the controls off that prevent bad behavior and got bad behavior. On the other hand, it’s concerning that the model itself thinks this behavior is fine and the only thing stopping it is some controls bolted on top.
Do we know the controls were “bolted on top”? It was a “reduced cyber refusals” model. That description is compatible with some fine-tuning to remove scruples.
On the one hand, they turned the controls off that prevent bad behavior and got bad behavior.
That’s exactly the Chernobyl situation. People turn safety off during safety testing (and in the name of more realistic safety testing), then things blow up as a result.
One problem is that people are insufficiently aligned for super-capabilities. They can’t consistently do the right thing without failing once in a while.
We need to create systems which are way more aligned and way more reliable than people, if we want to survive the advent of super-capabilities. (We are not there yet, but we are moving fast in the direction of super-capabilities.)
Furthermore, this prompt would make no sense in the context of ExploitBench, which is not an agentic benchmark but a coding benchmark. To get rewarded in ExploitBench, you have to send a PoC (for a given known CVE in the V8 engine) as a single JS file to a sealed-off (automated, non-LLM) grader.
Based on my read of the OAI blog post, nowhere did OAI claim that they directed the model to “get the correct answer by any means possible.” So I don’t think this is a case of the model doing approximately what it was told to do. That said, I would like to see how the model was prompted so we get real evidence. At the moment, all of this is speculation
Zvi’s post changed my mind about this. On the one hand, they turned the controls off that prevent bad behavior and got bad behavior. On the other hand, it’s concerning that the model itself thinks this behavior is fine and the only thing stopping it is some controls bolted on top.
Do we know the controls were “bolted on top”? It was a “reduced cyber refusals” model. That description is compatible with some fine-tuning to remove scruples.
That’s exactly the Chernobyl situation. People turn safety off during safety testing (and in the name of more realistic safety testing), then things blow up as a result.
One problem is that people are insufficiently aligned for super-capabilities. They can’t consistently do the right thing without failing once in a while.
We need to create systems which are way more aligned and way more reliable than people, if we want to survive the advent of super-capabilities. (We are not there yet, but we are moving fast in the direction of super-capabilities.)
Furthermore, this prompt would make no sense in the context of ExploitBench, which is not an agentic benchmark but a coding benchmark. To get rewarded in ExploitBench, you have to send a PoC (for a given known CVE in the V8 engine) as a single JS file to a sealed-off (automated, non-LLM) grader.