I’d think having a human produce or evaluate each answer would limit the training set far too much to be effective. It’s a large alignment tax I don’t expect labs to pay. Training data volume is crucial for capabilities.
I read your “neuralese is safer” post and thought it was interesting; I don’t remember this as a major point. If that’s a key consideration for the safety of neuralese, I’m afraid that’s not practical.
But I think AI reward estimation could be made much more resistant to this type of overclaiming success. I think this probably is mostly soluble, which may be enough. This is a nontrivial issue but not my first worry for alignment.
Thanks for the thoughts. Training a reward model is something I’d expect to require human data also. Is your picture here that training the reward model still requires human labels to train, but fewer of them (and then outputs from the reward model can be scaled easily), or that labs are directly re-purposing a standard trained model as a reward model with relatively little modification?[1]
Scraped data is cheap, and it’s not clear to me that there’s nothing one can do with it. I think the main restriction here (manual effort invested or not) is that the set of possible answers can’t be too large in a certain sense. Entropy coming from different phrasings of the same basic answer doesn’t count. But some tasks are fundamentally high-entropy, like coming up with a short story idea. That’s hard to train because we’re basically asking for the chain of thought to think of all the story ideas at once. If it just focuses on one, well that probably won’t be in the dataset, let alone the one that happened to be drawn for this SGD step. But trying to think of all short story ideas at once is not really an effective way to reason, I’m pretty sure.
Of course, even in the first case the reward model would be a fine-tune of some pretrained weights. The question is how much data is actually used to tune it.
I’d think having a human produce or evaluate each answer would limit the training set far too much to be effective. It’s a large alignment tax I don’t expect labs to pay. Training data volume is crucial for capabilities.
I read your “neuralese is safer” post and thought it was interesting; I don’t remember this as a major point. If that’s a key consideration for the safety of neuralese, I’m afraid that’s not practical.
But I think AI reward estimation could be made much more resistant to this type of overclaiming success. I think this probably is mostly soluble, which may be enough. This is a nontrivial issue but not my first worry for alignment.
Thanks for the thoughts. Training a reward model is something I’d expect to require human data also. Is your picture here that training the reward model still requires human labels to train, but fewer of them (and then outputs from the reward model can be scaled easily), or that labs are directly re-purposing a standard trained model as a reward model with relatively little modification? [1]
Scraped data is cheap, and it’s not clear to me that there’s nothing one can do with it. I think the main restriction here (manual effort invested or not) is that the set of possible answers can’t be too large in a certain sense. Entropy coming from different phrasings of the same basic answer doesn’t count. But some tasks are fundamentally high-entropy, like coming up with a short story idea. That’s hard to train because we’re basically asking for the chain of thought to think of all the story ideas at once. If it just focuses on one, well that probably won’t be in the dataset, let alone the one that happened to be drawn for this SGD step. But trying to think of all short story ideas at once is not really an effective way to reason, I’m pretty sure.
What is your first worry for alignment?
Of course, even in the first case the reward model would be a fine-tune of some pretrained weights. The question is how much data is actually used to tune it.