yep, I don’t think the distinction matters all that much—just a difference in what point you fork model training, but seems like the later you do it, the cheaper it will be for you and the more reflective your evals will be of your safety trained model
I think it does matter for emergent misalignment purposes. I’d think the emergent misalignment is more likely when you teach the model “these things are bad, don’t do them” then fine tune them to do one of those previously prohibited behaviors.
ah okay from openai’s persona features control EM paper, the helpful-only model they used was just trained on a standard post-training dataset with refusals + safety stuff filtered out: https://arxiv.org/pdf/2506.19823. Unclear whether they are still created this way though (or if there is any emergent refusal from subliminal learning on an synthetic dataset)
Or was helpful-only from the start.
yep, I don’t think the distinction matters all that much—just a difference in what point you fork model training, but seems like the later you do it, the cheaper it will be for you and the more reflective your evals will be of your safety trained model
I think it does matter for emergent misalignment purposes. I’d think the emergent misalignment is more likely when you teach the model “these things are bad, don’t do them” then fine tune them to do one of those previously prohibited behaviors.
ah okay from openai’s persona features control EM paper, the helpful-only model they used was just trained on a standard post-training dataset with refusals + safety stuff filtered out: https://arxiv.org/pdf/2506.19823. Unclear whether they are still created this way though (or if there is any emergent refusal from subliminal learning on an synthetic dataset)