Ironically pre-training data describing misaligned AI (a lot of which has been written by AI safety researchers) increases misalignment in LLMs which the authors of this paper call “self-fulfilling misalignment”: https://arxiv.org/pdf/2601.10160
This is not actually the finding in the paper! The relevant figure is Figure 4, in which up-sampling misalignment discussion causes a slight decrease in misaligned behaviour after DPO has been applied. The real relationship seems weird and U-shaped (no discussion is slightly better than some discussion, but lots of discussion is better still).
The robust result is that discussion of aligned behaviour definitely does reduce misalignment. Discussion of misaligned behaviour makes it a more salient concept, which actually gives a better handle for DPO to grab onto to reduce rates of misalignment. Same for discussion of aligned behaviour.
I wrote my original comment based on the abstract of the paper which doesn’t tell the whole story. After carefully reading the whole paper I agree with your comment.
Ironically pre-training data describing misaligned AI (a lot of which has been written by AI safety researchers) increases misalignment in LLMs which the authors of this paper call “self-fulfilling misalignment”: https://arxiv.org/pdf/2601.10160
This is not actually the finding in the paper! The relevant figure is Figure 4, in which up-sampling misalignment discussion causes a slight decrease in misaligned behaviour after DPO has been applied. The real relationship seems weird and U-shaped (no discussion is slightly better than some discussion, but lots of discussion is better still).
The robust result is that discussion of aligned behaviour definitely does reduce misalignment. Discussion of misaligned behaviour makes it a more salient concept, which actually gives a better handle for DPO to grab onto to reduce rates of misalignment. Same for discussion of aligned behaviour.
I wrote my original comment based on the abstract of the paper which doesn’t tell the whole story. After carefully reading the whole paper I agree with your comment.