It’s changing the dataset, but it’s doing so in an especially easy way:
You just add more datapoints. You don’t need to filter or relabel any of your existing data.
It uses datapoints—tied pairs—that many labs already collect but don’t use.
It seems fairly easy to generate more tied pairs, e.g. take any response and ask an LM to change it in lots of ways without changing its quality.
The responses making up tied pairs can come from the very same distribution as the rest of your data. Causal and spurious features might be tightly correlated on this distribution, but that doesn’t matter. You don’t need to do any decorrelating of these features. Tie training shrinks spurious weights regardless, including the weights on spurious features that aren’t even on your radar (so long as some of your tied pairs happen to differ in those features).
Is there some practical example you can give where this would deal with e.g. the spurious correlation between “This outcome looks good to a fallible human judge” and reward, which would nudge the generalization back towards “This outcome actually is good”?
Yep, imagine you have a set of prompts, each with a set of potential responses. Each response has an ‘actually good’ score and a ‘looks good’ score. These scores are correlated. If you do ordinary DPO/RLHF, the AI is going to put weight on the ‘looks good’ score, even if your preference data perfectly matches the actual goodness of responses, and even in the infinite-data limit (given the assumptions we mention in the post). You can shrink the weight on the ‘looks good’ score with tie training: collecting pairs of responses that are equally actually good and running DPO/RLHF with random/two-way labels.
Of course, depending on the domain, it might be difficult to collect pairs of responses that are equally actually good. Tie training doesn’t help with that task. It’s aimed at addressing goal misgeneralization, not reward misspecification. But even there:
You don’t need exact equal goodness for tie training to work. Training on near-ties works nearly as well. So it’s okay if judges are led a little astray by a response looking good.[1]
You could shrink the AI’s weight on the ‘looks good’ score by doing tie training in domains where you can be pretty confident of equal actual goodness (e.g. coding, math, etc.). This might lead the AI to put less weight on the ‘looks good’ score in other domains too.
In hard-to-verify domains, you could construct ties by giving an LM some instruction like ‘Change this response in lots of ways without changing its substance.’ (e.g. make sure it remains the same philosophical theory, or the same long-term forecast, or the same strategic recommendation, or the same AI safety proposal, etc.). That would give you a (near-)tie if the LM can achieve that. The analogue for ordinary DPO/RLHF—‘Change this response in lots of ways while making it better/worse’—seems significantly harder, and has other disadvantages that I can get into.
Compare to ordinary DPO/RLHF, where judges being led a little astray can be a serious problem:
Slight-preference training would also be more labor-intensive, because it takes work to determine which of two nearly-equal actions is truly better. And it would be risky too: ‘which of these near-equals is truly better?’ is exactly the sort of judgment that spurious features are likely to corrupt. If your judge is even slightly affected by spurious features, the winners in your preference data are likely to lean high-spurious, in which case your slight-preference training can backfire, increasing the AI’s spurious weights.
This isn’t quite the threat model I normally think about when discussing these kinds of problems. I imagine that we have two variables: looks-good-low-effort and is-good, which are correlated at some pretty high level on a non-pathological dataset, say 0.8. The dataset is then labelled according to looks-good-low-effort. The AI learns to put ~all its weight on learns-good-low-effort because that’s the best possible predictor.
What you want is some set of data which ties on is-good but varies on looks-good-low-effort. Unfortunately, you don’t have access to is-good, but you can access looks-good-high-effort which is correlated at a higher level, say 0.95, with is-good, with the resulting error very strongly correlated with that of looks-good-low-effort. So you produce some items which tie on looks-good-high-effort, and so presumably approximately tie on is-good, but have more noise on looks-good-low-effort.
Cross-domain split also seems like a problem.
I think this maybe makes sense, but I’d like a real test of it before drawing conclusions.
Yep, that all sounds right to me! If you can only access looks-good-high-effort, tie training can’t take you beyond that, but it will shrink the weight on looks-good-low-effort (along with any other spurious features that happen to differ across your pairs).
It’s changing the dataset, but it’s doing so in an especially easy way:
You just add more datapoints. You don’t need to filter or relabel any of your existing data.
It uses datapoints—tied pairs—that many labs already collect but don’t use.
It seems fairly easy to generate more tied pairs, e.g. take any response and ask an LM to change it in lots of ways without changing its quality.
The responses making up tied pairs can come from the very same distribution as the rest of your data. Causal and spurious features might be tightly correlated on this distribution, but that doesn’t matter. You don’t need to do any decorrelating of these features. Tie training shrinks spurious weights regardless, including the weights on spurious features that aren’t even on your radar (so long as some of your tied pairs happen to differ in those features).
Yep, imagine you have a set of prompts, each with a set of potential responses. Each response has an ‘actually good’ score and a ‘looks good’ score. These scores are correlated. If you do ordinary DPO/RLHF, the AI is going to put weight on the ‘looks good’ score, even if your preference data perfectly matches the actual goodness of responses, and even in the infinite-data limit (given the assumptions we mention in the post). You can shrink the weight on the ‘looks good’ score with tie training: collecting pairs of responses that are equally actually good and running DPO/RLHF with random/two-way labels.
Of course, depending on the domain, it might be difficult to collect pairs of responses that are equally actually good. Tie training doesn’t help with that task. It’s aimed at addressing goal misgeneralization, not reward misspecification. But even there:
You don’t need exact equal goodness for tie training to work. Training on near-ties works nearly as well. So it’s okay if judges are led a little astray by a response looking good.[1]
You could shrink the AI’s weight on the ‘looks good’ score by doing tie training in domains where you can be pretty confident of equal actual goodness (e.g. coding, math, etc.). This might lead the AI to put less weight on the ‘looks good’ score in other domains too.
In hard-to-verify domains, you could construct ties by giving an LM some instruction like ‘Change this response in lots of ways without changing its substance.’ (e.g. make sure it remains the same philosophical theory, or the same long-term forecast, or the same strategic recommendation, or the same AI safety proposal, etc.). That would give you a (near-)tie if the LM can achieve that. The analogue for ordinary DPO/RLHF—‘Change this response in lots of ways while making it better/worse’—seems significantly harder, and has other disadvantages that I can get into.
Compare to ordinary DPO/RLHF, where judges being led a little astray can be a serious problem:
This isn’t quite the threat model I normally think about when discussing these kinds of problems. I imagine that we have two variables: looks-good-low-effort and is-good, which are correlated at some pretty high level on a non-pathological dataset, say 0.8. The dataset is then labelled according to looks-good-low-effort. The AI learns to put ~all its weight on learns-good-low-effort because that’s the best possible predictor.
What you want is some set of data which ties on is-good but varies on looks-good-low-effort. Unfortunately, you don’t have access to is-good, but you can access looks-good-high-effort which is correlated at a higher level, say 0.95, with is-good, with the resulting error very strongly correlated with that of looks-good-low-effort. So you produce some items which tie on looks-good-high-effort, and so presumably approximately tie on is-good, but have more noise on looks-good-low-effort.
Cross-domain split also seems like a problem.
I think this maybe makes sense, but I’d like a real test of it before drawing conclusions.
Yep, that all sounds right to me! If you can only access looks-good-high-effort, tie training can’t take you beyond that, but it will shrink the weight on looks-good-low-effort (along with any other spurious features that happen to differ across your pairs).