Hmm, I mostly disagree. I do pay attention to capabilities research, and I feel it mostly doesn’t matter that much. Like I think if you know
The transformer architecture
What pretraining is
Roughly what scaling laws are
What gradient descent is
What SFT is
Roughly what RL is
What CoT is
You capture maybe 80%, unless you’re
Doing interp
Trying to forecast AI capabilities
Like, do you have any other examples than OPSD? Because other examples of capabilities research like MoEs linear attention variants, or new optimizers, strike me as obviously not changing the strategic picture much.
OPSD could be an example that does, but I’m not sure if I buy your argument. Seems to me OPSD alone does not scale. And to the degree it does scale, is no longer solving the problematic reward hacking we get from standard RLVR.
I’m not really confident, but my reasoning is, fundamentally, you have a bunch of environments with good and bad solutions. And what you’re doing is optimizing your model to score high on those metrics. And the space of high-scoring solutions is determined by things like whether your envs are hackable, not the optimization process you use.
Like, you can’t write human-supplied hints for all the hard tasks you want to train on.
So you’d have a model try until it solves it. Then use that trajectory to generate hints.
But that model might reward hack, and then the hint extracted from the solution might just be a hint on how to reward hack successfully.
Then you might have a model try to block reward hacky hints. But that’s not really very different from having a model that reads a trajectory and assigns −10 reward if it think the trajectory is reward-hacky.
Where is my reasoning wrong?
Some people even initially believed that this was a good thing for the mech interp faction!
Was this me? I still believe this I think? Or Neel Nanda said he thought it was not true, so I have to update on that outside-view a fair bit, but I still don’t really get why its bad.
Hmm, I mostly disagree. I do pay attention to capabilities research, and I feel it mostly doesn’t matter that much. Like I think if you know
The transformer architecture
What pretraining is
Roughly what scaling laws are
What gradient descent is
What SFT is
Roughly what RL is
What CoT is
You capture maybe 80%, unless you’re
Doing interp
Trying to forecast AI capabilities
Like, do you have any other examples than OPSD? Because other examples of capabilities research like MoEs linear attention variants, or new optimizers, strike me as obviously not changing the strategic picture much.
OPSD could be an example that does, but I’m not sure if I buy your argument. Seems to me OPSD alone does not scale. And to the degree it does scale, is no longer solving the problematic reward hacking we get from standard RLVR.
I’m not really confident, but my reasoning is, fundamentally, you have a bunch of environments with good and bad solutions. And what you’re doing is optimizing your model to score high on those metrics. And the space of high-scoring solutions is determined by things like whether your envs are hackable, not the optimization process you use.
Like, you can’t write human-supplied hints for all the hard tasks you want to train on.
So you’d have a model try until it solves it. Then use that trajectory to generate hints.
But that model might reward hack, and then the hint extracted from the solution might just be a hint on how to reward hack successfully.
Then you might have a model try to block reward hacky hints. But that’s not really very different from having a model that reads a trajectory and assigns −10 reward if it think the trajectory is reward-hacky.
Where is my reasoning wrong?
Was this me? I still believe this I think? Or Neel Nanda said he thought it was not true, so I have to update on that outside-view a fair bit, but I still don’t really get why its bad.