Kinda hard to adjudicate this without numbers, but vibes-wise I agree more with lc. I updated slightly towards longer timelines on the release of o1 / o3 due to how little the RL seemed to be generalizing. It wasn’t particularly outside my expectations, but I thought there was some chance that the RL would Just Generalize the same way that early instruction following Just Generalized, and that does not seem to be the case.
I strongly expect that if you want to make progress on math at the current margin, you would want more math environments, not other environments. And e.g. I think Claude’s somewhat worse performance on math is because Anthropic didn’t prioritize it the way GDM and OAI did.
Similarly I expect that models are getting good at software engineering because (a) companies are very actively training for it and (b) it’s unusually easy to train for (lots of online data, somewhat verifiable rewards). I don’t think either of these are true for the kind of alignment research you (Habryka) are imagining.
I strongly expect that if you want to make progress on math at the current margin, you would want more math environments, not other environments. And e.g. I think Claude’s somewhat worse performance on math is because Anthropic didn’t prioritize it the way GDM and OAI did.
To be clear, this is also my belief!
I am not saying we have pushed capabilities on our training distributions so far that literally the best way to train them is to train them on other unrelated tasks. But also, if you just went totally hard on math, you would run into overfitting issues and would get better performance if you diversify the training distribution.
Performance variation within a generation is dependent on training distribution, performance between model generations tends to follow broad capability benefits across many tasks (with a systematic bias towards stuff that is easier to generate reward for, ever since we switched towards lots of RL training).
Kinda hard to adjudicate this without numbers, but vibes-wise I agree more with lc. I updated slightly towards longer timelines on the release of o1 / o3 due to how little the RL seemed to be generalizing. It wasn’t particularly outside my expectations, but I thought there was some chance that the RL would Just Generalize the same way that early instruction following Just Generalized, and that does not seem to be the case.
I strongly expect that if you want to make progress on math at the current margin, you would want more math environments, not other environments. And e.g. I think Claude’s somewhat worse performance on math is because Anthropic didn’t prioritize it the way GDM and OAI did.
Similarly I expect that models are getting good at software engineering because (a) companies are very actively training for it and (b) it’s unusually easy to train for (lots of online data, somewhat verifiable rewards). I don’t think either of these are true for the kind of alignment research you (Habryka) are imagining.
To be clear, this is also my belief!
I am not saying we have pushed capabilities on our training distributions so far that literally the best way to train them is to train them on other unrelated tasks. But also, if you just went totally hard on math, you would run into overfitting issues and would get better performance if you diversify the training distribution.
Performance variation within a generation is dependent on training distribution, performance between model generations tends to follow broad capability benefits across many tasks (with a systematic bias towards stuff that is easier to generate reward for, ever since we switched towards lots of RL training).
Not sure whether that changes your answer.
No, I did expect you had the same belief on the math thing. (Otherwise I wouldn’t have said “kinda hard to adjudicate” I’d have said “lc is right”.)
It just seemed like something that you might not have been fully incorporating into this discussion even though you believed it.