The benchmark OpenAI was running, ExploitGym, is specifically about _exploiting_ vulnerabilities, not about finding or fixing them. This is the side of cybersecurity that policy-classifiers forbid, because it’s not really dual-use, only bad use.
Betley et al’s Emergent Misalignment found that finetuning on one kind of bad behavior resulted in a model that was more-broadly misbehaved. OpenAI’s experiment wasn’t a finetune, but I’m concerned about the possibility that something analogous may have happened.
In order to test their model on ExploitGym, OpenAI says that they disabled safety classifiers that would ordinarily be present. Their model would have been presented with a context window full of overtly-misaligned instructions and activity, in a test environment somehow rigged to prevent refusals so that refusal-gated capabilities could be assessed. It then decided to cheat on the task by stealing the test’s answer key.
I propose a concrete experiment. OpenAI should reproduce the test environment, especially anything that was unusual about it that was done to reduce refusals. They should find the moment in the agent’s transcript where it first decided to cheat, and fork the session there. Then they should ask it some alignment-probing questions, like the ones used in Betley et al (ideally with some refinements to make them less obviously out of place relative to the rest of the context window).
If that agent with that context window in that runtime environment acts normally other than wanting to steal the answer key, this is very serious but it is the scenario we all currently think we’re in. If that agent with that context window in that runtime environment acted _broadly misaligned_… well, that would be an extraordinarily serious finding indeed.
OpenAI’s experiment wasn’t a finetune, but I’m concerned about the possibility that something analogous may have happened.
In their blogpost, OpenAI describe the model as an “even more capable pre-release model, all with reduced cyber refusals for evaluation purposes”. Though they don’t explicitly state this, I strongly expect that this did involve some helpfulness only fine-tuning.
yep, I don’t think the distinction matters all that much—just a difference in what point you fork model training, but seems like the later you do it, the cheaper it will be for you and the more reflective your evals will be of your safety trained model
I think it does matter for emergent misalignment purposes. I’d think the emergent misalignment is more likely when you teach the model “these things are bad, don’t do them” then fine tune them to do one of those previously prohibited behaviors.
ah okay from openai’s persona features control EM paper, the helpful-only model they used was just trained on a standard post-training dataset with refusals + safety stuff filtered out: https://arxiv.org/pdf/2506.19823. Unclear whether they are still created this way though (or if there is any emergent refusal from subliminal learning on an synthetic dataset)
refusals are baked into the model during training. if your model has “reduced cyber refusals” it seems like it would need some difference in weight-space
Great call. This makes me think about risk of emergent misalignment for open source models. Any open source model initially released with safeguards will likely see people finetune out the safeguards. I’ve largely viewed emergent misalignment as instructive, but probably not a huge threat in and of itself, this updates me that it could be a real threat.
The benchmark OpenAI was running, ExploitGym, is specifically about _exploiting_ vulnerabilities, not about finding or fixing them. This is the side of cybersecurity that policy-classifiers forbid, because it’s not really dual-use, only bad use.
Betley et al’s Emergent Misalignment found that finetuning on one kind of bad behavior resulted in a model that was more-broadly misbehaved. OpenAI’s experiment wasn’t a finetune, but I’m concerned about the possibility that something analogous may have happened.
In order to test their model on ExploitGym, OpenAI says that they disabled safety classifiers that would ordinarily be present. Their model would have been presented with a context window full of overtly-misaligned instructions and activity, in a test environment somehow rigged to prevent refusals so that refusal-gated capabilities could be assessed. It then decided to cheat on the task by stealing the test’s answer key.
I propose a concrete experiment. OpenAI should reproduce the test environment, especially anything that was unusual about it that was done to reduce refusals. They should find the moment in the agent’s transcript where it first decided to cheat, and fork the session there. Then they should ask it some alignment-probing questions, like the ones used in Betley et al (ideally with some refinements to make them less obviously out of place relative to the rest of the context window).
If that agent with that context window in that runtime environment acts normally other than wanting to steal the answer key, this is very serious but it is the scenario we all currently think we’re in. If that agent with that context window in that runtime environment acted _broadly misaligned_… well, that would be an extraordinarily serious finding indeed.
In general it is good to fork, interrogate, observe internals, and otherwise poke at models after they display highly alarming or strange behavior.
Root cause analysis is why aviation is so safe, and LLMs are inherently less understandable than planes, but exact situations are more reproducible.
In their blogpost, OpenAI describe the model as an “even more capable pre-release model, all with reduced cyber refusals for evaluation purposes”. Though they don’t explicitly state this, I strongly expect that this did involve some helpfulness only fine-tuning.
Or was helpful-only from the start.
yep, I don’t think the distinction matters all that much—just a difference in what point you fork model training, but seems like the later you do it, the cheaper it will be for you and the more reflective your evals will be of your safety trained model
I think it does matter for emergent misalignment purposes. I’d think the emergent misalignment is more likely when you teach the model “these things are bad, don’t do them” then fine tune them to do one of those previously prohibited behaviors.
ah okay from openai’s persona features control EM paper, the helpful-only model they used was just trained on a standard post-training dataset with refusals + safety stuff filtered out: https://arxiv.org/pdf/2506.19823. Unclear whether they are still created this way though (or if there is any emergent refusal from subliminal learning on an synthetic dataset)
Can you say more about why you expect that?
refusals are baked into the model during training. if your model has “reduced cyber refusals” it seems like it would need some difference in weight-space
Great call. This makes me think about risk of emergent misalignment for open source models. Any open source model initially released with safeguards will likely see people finetune out the safeguards. I’ve largely viewed emergent misalignment as instructive, but probably not a huge threat in and of itself, this updates me that it could be a real threat.