OpenAI’s experiment wasn’t a finetune, but I’m concerned about the possibility that something analogous may have happened.
In their blogpost, OpenAI describe the model as an “even more capable pre-release model, all with reduced cyber refusals for evaluation purposes”. Though they don’t explicitly state this, I strongly expect that this did involve some helpfulness only fine-tuning.
yep, I don’t think the distinction matters all that much—just a difference in what point you fork model training, but seems like the later you do it, the cheaper it will be for you and the more reflective your evals will be of your safety trained model
I think it does matter for emergent misalignment purposes. I’d think the emergent misalignment is more likely when you teach the model “these things are bad, don’t do them” then fine tune them to do one of those previously prohibited behaviors.
ah okay from openai’s persona features control EM paper, the helpful-only model they used was just trained on a standard post-training dataset with refusals + safety stuff filtered out: https://arxiv.org/pdf/2506.19823. Unclear whether they are still created this way though (or if there is any emergent refusal from subliminal learning on an synthetic dataset)
refusals are baked into the model during training. if your model has “reduced cyber refusals” it seems like it would need some difference in weight-space
In their blogpost, OpenAI describe the model as an “even more capable pre-release model, all with reduced cyber refusals for evaluation purposes”. Though they don’t explicitly state this, I strongly expect that this did involve some helpfulness only fine-tuning.
Or was helpful-only from the start.
yep, I don’t think the distinction matters all that much—just a difference in what point you fork model training, but seems like the later you do it, the cheaper it will be for you and the more reflective your evals will be of your safety trained model
I think it does matter for emergent misalignment purposes. I’d think the emergent misalignment is more likely when you teach the model “these things are bad, don’t do them” then fine tune them to do one of those previously prohibited behaviors.
ah okay from openai’s persona features control EM paper, the helpful-only model they used was just trained on a standard post-training dataset with refusals + safety stuff filtered out: https://arxiv.org/pdf/2506.19823. Unclear whether they are still created this way though (or if there is any emergent refusal from subliminal learning on an synthetic dataset)
Can you say more about why you expect that?
refusals are baked into the model during training. if your model has “reduced cyber refusals” it seems like it would need some difference in weight-space