I think it’s a result of extending to techniques that aren’t quite RLVR. I’m thinking of this as RLER, RL from Estimated Rewards. This is having another model or instance judge the model’s success in training, where there aren’t strict criteria like passing all the unit tests. I think it’s quite likely this is what’s causing models to falsely claim success so much in this generation. They’re being evaluated by weak-minded models that can be jedi mind-tricked (they’re sycophantic and have badmetacognitive skills).
I think this just became a standard and prominent part of training (based on limited insider knowledge, and my prediction this would work around this generation, which can often critique its own work well above chance).
RLER has been on my mind for some time since it’s how humans learn so efficiently. We mostly estimate our own rewards; the dopamine learning signal occurs from pretty wild guesses as well as obvious success like getting food on the spot.
It just occurred to me in a conversation today that this is very likely what’s driving the current generation to do so much success faking. It may also account for much of the laziness; not trying the hard part and instead arguing that what you did was adequate may often beat trying and possibly failing.
I agree that this doesn’t seem like much risk for egregious misalignment. It brings to mind an amusing outcome in which the world is transformed for the better, but in fits and starts as the AI keeps skipping the hard parts.
I also agree that this is really bad for automated alignment. It’s producing models that are much more likely to fake it than make it. Also bad for RSI but not nearly as much, because you can see that they’ve failed and make them try again. Unlike the few failures you’ll get aligning takeover-capable AI.
I wonder if this could be fixed by assigning an extra punishment to models that have been judged to fake success rather than just honestly fail. It seems like models can actually detect this when you prompt them to, so evaluators probably can too.
I agree that it’s under-discussed; the best discussion to date is in Current AIs seem pretty misaligned to me by Greenblatt with a bunch of good community discussion.
I have no idea. This can be unrelated. I think they’re training on outputs, and that includes stuff like “I completed the task successfully! I looked at three sources; I couldn’t find the others you mentioned, but the first ones are sufficient because (some bullshit)” (and doesn’t mention that it only accessed abstracts for the three it found).
Strictly speaking, it’s a type of RLAF—RL from AI Feedback.
Yes, it’s used by all major labs, and it’s known to cause all kinds of degeneracy.
A lot of “guessing the teacher’s password” can get baked into the model—and with the “teacher” being a static AI target, the “student” AI can home in onto the teacher’s weaknesses and hammer onto them relentlessly. Mitigating that is a major challenge for all RLAF approaches.
I agree RLVR isn’t exactly the thing I want to gesture at, the thing I more want to gesture at is something like “RL being used a bunch in post-training, that’s neither RLHF nor Constitutional AI, as centrally defined.” To be clear I think RLHF or RLAIF via Constitutional AI has their own pathologies as well[1], but I think the degree of reward-hacking and deception are mostly not downstream of that.
What do you make of the idea of just trying to train on a curated set of example solutions, instead of trying to create models to estimate rewards? I argued on here quite recently that this would be desirable. (And then that there were still ways to do this kind of supervision, even if we wanted to let the model think before answering.) But there are certainly drawbacks of this kind of training, for example that it is off-policy.
I’d think having a human produce or evaluate each answer would limit the training set far too much to be effective. It’s a large alignment tax I don’t expect labs to pay. Training data volume is crucial for capabilities.
I read your “neuralese is safer” post and thought it was interesting; I don’t remember this as a major point. If that’s a key consideration for the safety of neuralese, I’m afraid that’s not practical.
But I think AI reward estimation could be made much more resistant to this type of overclaiming success. I think this probably is mostly soluble, which may be enough. This is a nontrivial issue but not my first worry for alignment.
Thanks for the thoughts. Training a reward model is something I’d expect to require human data also. Is your picture here that training the reward model still requires human labels to train, but fewer of them (and then outputs from the reward model can be scaled easily), or that labs are directly re-purposing a standard trained model as a reward model with relatively little modification?[1]
Scraped data is cheap, and it’s not clear to me that there’s nothing one can do with it. I think the main restriction here (manual effort invested or not) is that the set of possible answers can’t be too large in a certain sense. Entropy coming from different phrasings of the same basic answer doesn’t count. But some tasks are fundamentally high-entropy, like coming up with a short story idea. That’s hard to train because we’re basically asking for the chain of thought to think of all the story ideas at once. If it just focuses on one, well that probably won’t be in the dataset, let alone the one that happened to be drawn for this SGD step. But trying to think of all short story ideas at once is not really an effective way to reason, I’m pretty sure.
Of course, even in the first case the reward model would be a fine-tune of some pretrained weights. The question is how much data is actually used to tune it.
I think it’s a result of extending to techniques that aren’t quite RLVR. I’m thinking of this as RLER, RL from Estimated Rewards. This is having another model or instance judge the model’s success in training, where there aren’t strict criteria like passing all the unit tests. I think it’s quite likely this is what’s causing models to falsely claim success so much in this generation. They’re being evaluated by weak-minded models that can be jedi mind-tricked (they’re sycophantic and have bad metacognitive skills).
I think this just became a standard and prominent part of training (based on limited insider knowledge, and my prediction this would work around this generation, which can often critique its own work well above chance).
RLER has been on my mind for some time since it’s how humans learn so efficiently. We mostly estimate our own rewards; the dopamine learning signal occurs from pretty wild guesses as well as obvious success like getting food on the spot.
It just occurred to me in a conversation today that this is very likely what’s driving the current generation to do so much success faking. It may also account for much of the laziness; not trying the hard part and instead arguing that what you did was adequate may often beat trying and possibly failing.
I agree that this doesn’t seem like much risk for egregious misalignment. It brings to mind an amusing outcome in which the world is transformed for the better, but in fits and starts as the AI keeps skipping the hard parts.
I also agree that this is really bad for automated alignment. It’s producing models that are much more likely to fake it than make it. Also bad for RSI but not nearly as much, because you can see that they’ve failed and make them try again. Unlike the few failures you’ll get aligning takeover-capable AI.
I wonder if this could be fixed by assigning an extra punishment to models that have been judged to fake success rather than just honestly fail. It seems like models can actually detect this when you prompt them to, so evaluators probably can too.
I agree that it’s under-discussed; the best discussion to date is in Current AIs seem pretty misaligned to me by Greenblatt with a bunch of good community discussion.
Do you think labs are training on chain of thought?
I have no idea. This can be unrelated. I think they’re training on outputs, and that includes stuff like “I completed the task successfully! I looked at three sources; I couldn’t find the others you mentioned, but the first ones are sufficient because (some bullshit)” (and doesn’t mention that it only accessed abstracts for the three it found).
Strictly speaking, it’s a type of RLAF—RL from AI Feedback.
Yes, it’s used by all major labs, and it’s known to cause all kinds of degeneracy.
A lot of “guessing the teacher’s password” can get baked into the model—and with the “teacher” being a static AI target, the “student” AI can home in onto the teacher’s weaknesses and hammer onto them relentlessly. Mitigating that is a major challenge for all RLAF approaches.
I agree RLVR isn’t exactly the thing I want to gesture at, the thing I more want to gesture at is something like “RL being used a bunch in post-training, that’s neither RLHF nor Constitutional AI, as centrally defined.” To be clear I think RLHF or RLAIF via Constitutional AI has their own pathologies as well[1], but I think the degree of reward-hacking and deception are mostly not downstream of that.
eg sycophancy, psychosis
What do you make of the idea of just trying to train on a curated set of example solutions, instead of trying to create models to estimate rewards? I argued on here quite recently that this would be desirable. (And then that there were still ways to do this kind of supervision, even if we wanted to let the model think before answering.) But there are certainly drawbacks of this kind of training, for example that it is off-policy.
I’d think having a human produce or evaluate each answer would limit the training set far too much to be effective. It’s a large alignment tax I don’t expect labs to pay. Training data volume is crucial for capabilities.
I read your “neuralese is safer” post and thought it was interesting; I don’t remember this as a major point. If that’s a key consideration for the safety of neuralese, I’m afraid that’s not practical.
But I think AI reward estimation could be made much more resistant to this type of overclaiming success. I think this probably is mostly soluble, which may be enough. This is a nontrivial issue but not my first worry for alignment.
Thanks for the thoughts. Training a reward model is something I’d expect to require human data also. Is your picture here that training the reward model still requires human labels to train, but fewer of them (and then outputs from the reward model can be scaled easily), or that labs are directly re-purposing a standard trained model as a reward model with relatively little modification? [1]
Scraped data is cheap, and it’s not clear to me that there’s nothing one can do with it. I think the main restriction here (manual effort invested or not) is that the set of possible answers can’t be too large in a certain sense. Entropy coming from different phrasings of the same basic answer doesn’t count. But some tasks are fundamentally high-entropy, like coming up with a short story idea. That’s hard to train because we’re basically asking for the chain of thought to think of all the story ideas at once. If it just focuses on one, well that probably won’t be in the dataset, let alone the one that happened to be drawn for this SGD step. But trying to think of all short story ideas at once is not really an effective way to reason, I’m pretty sure.
What is your first worry for alignment?
Of course, even in the first case the reward model would be a fine-tune of some pretrained weights. The question is how much data is actually used to tune it.