Thanks for the thoughts. Training a reward model is something I’d expect to require human data also. Is your picture here that training the reward model still requires human labels to train, but fewer of them (and then outputs from the reward model can be scaled easily), or that labs are directly re-purposing a standard trained model as a reward model with relatively little modification?[1]
Scraped data is cheap, and it’s not clear to me that there’s nothing one can do with it. I think the main restriction here (manual effort invested or not) is that the set of possible answers can’t be too large in a certain sense. Entropy coming from different phrasings of the same basic answer doesn’t count. But some tasks are fundamentally high-entropy, like coming up with a short story idea. That’s hard to train because we’re basically asking for the chain of thought to think of all the story ideas at once. If it just focuses on one, well that probably won’t be in the dataset, let alone the one that happened to be drawn for this SGD step. But trying to think of all short story ideas at once is not really an effective way to reason, I’m pretty sure.
Of course, even in the first case the reward model would be a fine-tune of some pretrained weights. The question is how much data is actually used to tune it.
Thanks for the thoughts. Training a reward model is something I’d expect to require human data also. Is your picture here that training the reward model still requires human labels to train, but fewer of them (and then outputs from the reward model can be scaled easily), or that labs are directly re-purposing a standard trained model as a reward model with relatively little modification? [1]
Scraped data is cheap, and it’s not clear to me that there’s nothing one can do with it. I think the main restriction here (manual effort invested or not) is that the set of possible answers can’t be too large in a certain sense. Entropy coming from different phrasings of the same basic answer doesn’t count. But some tasks are fundamentally high-entropy, like coming up with a short story idea. That’s hard to train because we’re basically asking for the chain of thought to think of all the story ideas at once. If it just focuses on one, well that probably won’t be in the dataset, let alone the one that happened to be drawn for this SGD step. But trying to think of all short story ideas at once is not really an effective way to reason, I’m pretty sure.
What is your first worry for alignment?
Of course, even in the first case the reward model would be a fine-tune of some pretrained weights. The question is how much data is actually used to tune it.