Thank you for writing this, Elliott. I’m on a team of folks wrapping up a project that is pretty similar to this.
Our team is composed of most of the folks behind Tice et al. (2026). In that work, we found that alignment pretraining does not help mitigate SFT-induced emergent misalignment when training on the risky-advice datasets of Turnet et al. (2025). My collaborators at Geodesic and I have been exploring follow-ups to Alignment Pretraining with the aim of making personas “adversarially robust” against post-training selection that inadvertently favours misaligned personas. Specifically, we’ve been looking into an intervention analogous to inoculation pretraining that explains away misaligned behaviour during training. We have so far had modestly positive results, but think this approach is not production-ready yet.
We’re aiming to share findings publicly with the community in the coming weeks.
Thank you for writing this, Elliott. I’m on a team of folks wrapping up a project that is pretty similar to this.
Our team is composed of most of the folks behind Tice et al. (2026). In that work, we found that alignment pretraining does not help mitigate SFT-induced emergent misalignment when training on the risky-advice datasets of Turnet et al. (2025). My collaborators at Geodesic and I have been exploring follow-ups to Alignment Pretraining with the aim of making personas “adversarially robust” against post-training selection that inadvertently favours misaligned personas. Specifically, we’ve been looking into an intervention analogous to inoculation pretraining that explains away misaligned behaviour during training. We have so far had modestly positive results, but think this approach is not production-ready yet.
We’re aiming to share findings publicly with the community in the coming weeks.
That’s great to hear! Looking forward to seeing the results.