Though, I wonder how is the oversampling approach different from a KL prior on the base model (the one without an inoculation prompt) for the mixed examples obtained after classification?
We tried that at some point. The downside is that KL on neutral data suppresses learning the desired traits. This is true even when the neutral data is “invariant” to the desired traits (i.e., desired traits should not impact that data).
Just so that I understand correctly, are you suggesting training on the DT+UT examples with the inoculation prompt, and also doing a second forward pass without the inoculation prompt but with KL term to regularise the distribution towards the frozen version of the base model?
My intuition is that it would help with suppressing UT, but would also hinder learning DT. It might also be the case that it is less of a problem for traits that the base model already has, say speaking French, and more problematic for traits far from the base distribution.
Looks like a reasonable extension! Cool!
Though, I wonder how is the oversampling approach different from a KL prior on the base model (the one without an inoculation prompt) for the mixed examples obtained after classification?
We tried that at some point. The downside is that KL on neutral data suppresses learning the desired traits. This is true even when the neutral data is “invariant” to the desired traits (i.e., desired traits should not impact that data).
Thank you!
Just so that I understand correctly, are you suggesting training on the DT+UT examples with the inoculation prompt, and also doing a second forward pass without the inoculation prompt but with KL term to regularise the distribution towards the frozen version of the base model?
My intuition is that it would help with suppressing UT, but would also hinder learning DT. It might also be the case that it is less of a problem for traits that the base model already has, say speaking French, and more problematic for traits far from the base distribution.