This is not actually the finding in the paper! The relevant figure is Figure 4, in which up-sampling misalignment discussion causes a slight decrease in misaligned behaviour after DPO has been applied. The real relationship seems weird and U-shaped (no discussion is slightly better than some discussion, but lots of discussion is better still).
The robust result is that discussion of aligned behaviour definitely does reduce misalignment. Discussion of misaligned behaviour makes it a more salient concept, which actually gives a better handle for DPO to grab onto to reduce rates of misalignment. Same for discussion of aligned behaviour.
I wrote my original comment based on the abstract of the paper which doesn’t tell the whole story. After carefully reading the whole paper I agree with your comment.
This is not actually the finding in the paper! The relevant figure is Figure 4, in which up-sampling misalignment discussion causes a slight decrease in misaligned behaviour after DPO has been applied. The real relationship seems weird and U-shaped (no discussion is slightly better than some discussion, but lots of discussion is better still).
The robust result is that discussion of aligned behaviour definitely does reduce misalignment. Discussion of misaligned behaviour makes it a more salient concept, which actually gives a better handle for DPO to grab onto to reduce rates of misalignment. Same for discussion of aligned behaviour.
I wrote my original comment based on the abstract of the paper which doesn’t tell the whole story. After carefully reading the whole paper I agree with your comment.