One theory is that this is effect only occurs in LoRAs. I haven’t thought about this much, but maybe post training on LoRAs leads to strong behavioral changes but weak deep knowledge / world model changes. (This might be dumb, I don’t know much about training on LoRAs vs full parameters.)
Interesting suggestion. Repeated negations interspersed into text is a pretty weird behavior for most Internet documents. We are a bit grasping at straws here, but then, this behavior IS really weird. So sure, someone should try it with full-parameter training.
Agree it’s worth trying! I’d be surprised if it changes things, but worth seeing what happens. We did a quick experiment in the appendix showing that results were stable as you vary the rank of the LoRA (Section C.3). However, we only test up to rank 64 (max when finetuning via Tinker).
Yup, saw that, and appreciate the thoroughness. As someone else remarked, you seem to have tried really hard to make this go away — the effort is impressive, and makes your result that this was surprisingly resilient even stronger.
One theory is that this is effect only occurs in LoRAs. I haven’t thought about this much, but maybe post training on LoRAs leads to strong behavioral changes but weak deep knowledge / world model changes. (This might be dumb, I don’t know much about training on LoRAs vs full parameters.)
I like your theory. It would be interesting to see some mechanistic interpretability studies of this phenomena.
Interesting suggestion. Repeated negations interspersed into text is a pretty weird behavior for most Internet documents. We are a bit grasping at straws here, but then, this behavior IS really weird. So sure, someone should try it with full-parameter training.
Agree it’s worth trying! I’d be surprised if it changes things, but worth seeing what happens. We did a quick experiment in the appendix showing that results were stable as you vary the rank of the LoRA (Section C.3). However, we only test up to rank 64 (max when finetuning via Tinker).
Yup, saw that, and appreciate the thoroughness. As someone else remarked, you seem to have tried really hard to make this go away — the effort is impressive, and makes your result that this was surprisingly resilient even stronger.