the leading LLM companies have said that they spend very little effort on RL-for-math
I’m curious as to the source for this. Even if true, it could be precisely because verifiability makes RL-for-math easy.
In the case of RL-for-math, the RL is mostly-or-entirely “RLAIF” (RL from AI Feedback), not “RLVR” (RL with Verifiable Rewards)”
Your evidence for this is the discussion on Deepmind’s IMO 2025 approach, however this is an inference-time scaffold and not RL-for-math.
That being said, the core idea—the correctness of imitation data is undervalued relative to verifiability—is interesting.
A few thoughts on this
It’s important to note that the two components are related—verifiability enables correct imitation data via synthetic data pipelines.
It can be helpful to have error-correction in pre-training data. See: Allen-Zhu’s Physics of LMs 2.2.
You might be interested in this paper that studies related questions: On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models.
I’m curious as to the source for this. Even if true, it could be precisely because verifiability makes RL-for-math easy.
Your evidence for this is the discussion on Deepmind’s IMO 2025 approach, however this is an inference-time scaffold and not RL-for-math.
That being said, the core idea—the correctness of imitation data is undervalued relative to verifiability—is interesting.
A few thoughts on this
It’s important to note that the two components are related—verifiability enables correct imitation data via synthetic data pipelines.
It can be helpful to have error-correction in pre-training data. See: Allen-Zhu’s Physics of LMs 2.2.
You might be interested in this paper that studies related questions: On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models.