Very nice. That’s been my intuition for a while, but I’d considered it somewhat compromised by the NVIDIA paper you mentioned. My counter-intuition was that pre-RLVR LLMs could solve every subproblem in a long problem 10 percent of the time, but for tricky enough problems, they need to do that enough times in succession that, in practice, the success rate was zero.
Yeah, my thoughts exactly (if I understand you correctly).
I mentioned in the OP that the NVIDIA paper (Liu et al. “ProRL”) says “RL can indeed discover genuinely new solution pathways entirely absent in base models, when given sufficient training time and applied to novel reasoning tasks,” but then added that I didn’t the paper had proved it. I didn’t explain in the OP why I was skeptical.
…But what I was thinking was: if solving the problem requires doing the right step 20 times in a row, and the base model has a 10% chance of taking the right step each time, then the base model will never succeed, at least not in the number of attempts that they could afford to try. But then if RLVR gets it from 10% to 95%, it will succeed a lot. But upping the probability from 10% to 95% is not what one would reasonably call a “genuinely new solution pathway entirely absent in the base model”.
(Warning: I skimmed the paper and might be misunderstanding how they were justifying that claim.)
Very nice. That’s been my intuition for a while, but I’d considered it somewhat compromised by the NVIDIA paper you mentioned. My counter-intuition was that pre-RLVR LLMs could solve every subproblem in a long problem 10 percent of the time, but for tricky enough problems, they need to do that enough times in succession that, in practice, the success rate was zero.
Yeah, my thoughts exactly (if I understand you correctly).
I mentioned in the OP that the NVIDIA paper (Liu et al. “ProRL”) says “RL can indeed discover genuinely new solution pathways entirely absent in base models, when given sufficient training time and applied to novel reasoning tasks,” but then added that I didn’t the paper had proved it. I didn’t explain in the OP why I was skeptical.
…But what I was thinking was: if solving the problem requires doing the right step 20 times in a row, and the base model has a 10% chance of taking the right step each time, then the base model will never succeed, at least not in the number of attempts that they could afford to try. But then if RLVR gets it from 10% to 95%, it will succeed a lot. But upping the probability from 10% to 95% is not what one would reasonably call a “genuinely new solution pathway entirely absent in the base model”.
(Warning: I skimmed the paper and might be misunderstanding how they were justifying that claim.)