I suspect this is the same effect as in the Phantom Transfer paper, although shown in a much cleaner framing. In that paper, we showed that you can take out all the samples which seem like they might induce some hidden generalization and, nonetheless, the generalization persists. My expectation for how this works is some intersection of your two hypotheses:
The behaviors are still “taught” during SFT, but the effect is very subliminal/spread out over all the data, i.e. is not really filterable. The behaviors are already existent in the midtrain base-model, perhaps in form of (many) assistant persona(s), and gets elicited via the SFT training
My guess is that:
The model exists in some weighted average of a set of ‘personas’ (for lack of a better word)
These personas contain in them both preferences/motivations and spurious formatting. I.e., each model behavior or formatting property is tied to some subset of these personas
When the model is SFT’ed on data which seems likely to be generated by one of these personas, it upweights the entire persona in its weighted average. I.e., if you teach the model the formatting, it generalizes the behaviors that go along with this formatting
That hypothesis would explain both the phantom transfer effects and the things you’re seeing here. In PT, it’s that the persona which gets upweighted has a preference for, e.g., Catholicism. In your case, it’s that the persona which gets upweighted has a preference for bolded text or liberal views.
The reason, then, that data filtering doesn’t work is that the set of features which are persona-inducing are smeared across the text in a bunch of uninterpretable ways.
curious if you agree/disagree! I’m also curious how one might falsify this hypothesis. Will think on it a bit.
We did take a quick look at phantom transfer at the start of this project, am personally more of the view that phantom transfer is caused by very subtle semantic effects. In the case of your example, I suspect that if we took a real catholicism expert, he/she would be able to identify that every document in the phantom transfer set slightly favors catholicism. Of course it could still be that the catholicism persona is getting up weighted of course!
Nice! This is really cool.
I suspect this is the same effect as in the Phantom Transfer paper, although shown in a much cleaner framing. In that paper, we showed that you can take out all the samples which seem like they might induce some hidden generalization and, nonetheless, the generalization persists. My expectation for how this works is some intersection of your two hypotheses:
My guess is that:
The model exists in some weighted average of a set of ‘personas’ (for lack of a better word)
These personas contain in them both preferences/motivations and spurious formatting. I.e., each model behavior or formatting property is tied to some subset of these personas
When the model is SFT’ed on data which seems likely to be generated by one of these personas, it upweights the entire persona in its weighted average. I.e., if you teach the model the formatting, it generalizes the behaviors that go along with this formatting
That hypothesis would explain both the phantom transfer effects and the things you’re seeing here. In PT, it’s that the persona which gets upweighted has a preference for, e.g., Catholicism. In your case, it’s that the persona which gets upweighted has a preference for bolded text or liberal views.
The reason, then, that data filtering doesn’t work is that the set of features which are persona-inducing are smeared across the text in a bunch of uninterpretable ways.
curious if you agree/disagree! I’m also curious how one might falsify this hypothesis. Will think on it a bit.
We did take a quick look at phantom transfer at the start of this project, am personally more of the view that phantom transfer is caused by very subtle semantic effects. In the case of your example, I suspect that if we took a real catholicism expert, he/she would be able to identify that every document in the phantom transfer set slightly favors catholicism. Of course it could still be that the catholicism persona is getting up weighted of course!
With the case of SFT—I think your hypothesis is reasonable. I really liked the experiments in this write up, trying to isolate behaviors that are unique to some models and try differently distilled SFT datasets to see if they remain, like was done here: https://www.lesswrong.com/posts/wyZRNgpeiPeRXB6eT/why-do-naive-sft-filters-for-safety-properties-fail