LLMs are (still) mostly powered by imitative learning, not RL
Reinforcement learning from verifiable rewards (RLVR) is the hot new thing in LLM training. It’s so hot, and people spend so much time talking about it, that they sometimes lose sight of the big picture.
Stepping back, LLMs can do lots of very impressive things. How? Where did those capabilities come from? Fundamentally, they come from a combination of:
(1) Imitative learning, including pretraining and supervised fine-tuning (SFT)
(2) Reinforcement learning, including RL from human feedback [RLHF], RL from AI feedback [RLAIF], and especially RLVR.[1]
If we look at the final trained LLM, we can ask how important each of those two pieces was, in explaining the LLM’s capabilities. And my claim is that it’s way more (1) than (2).
I’ll start in §1 with some relevant evidence, and then in §2 I’ll circle back to operationalizing exactly what I’m claiming, and finally in §3, three reasons why we should care—namely, it affects how we should think about chain-of-thought legibility, about LLM capabilities, and about LLM alignment.
Note that I am not arguing that RLVR does not importantly contribute to LLM capabilities. That would be absurd! Of course it does! Companies use RLVR because it works, and I expect them to continue doing so more and more. Again, the things I’m actually claiming are in §2–§3.
1. Some relevant evidence
1.1 Theoretically, each GPU-hour spent on RL should have orders of magnitude less contribution to LLM capabilities than a GPU-hour spent on imitative learning
The fraction of total LLM training compute that goes towards RLVR is high (one source guessed 20%; for all I know it’s >50%). But that’s different from how to divvy up the credit that pretraining/SFT versus RLVR can take for LLM capabilities. Why is that different? See:
“The Extreme Inefficiency of RL for Frontier Models” (Toby Ord 2025)
“RL is even more information inefficient than you thought” (Dwarkesh Patel 2025)
Based on those posts, if RLVR is 50% of the training compute, then it might only be 0.01% of the information content imparted by training. And unless the learning rates are drastically different, it would also be 0.01% of changes to the weights.
1.2 The chain-of-thought (CoT) is still obviously strongly influenced by imitative learning
People emphasize how frontier model CoTs can be pretty weird, and that’s true and important, but if you step back a bit, in the grand scheme of things, CoTs are still very obviously strongly influenced by imitative learning from human-created (and hence legible) text.
See for example,
I prefer to start from the opposite side. What would a really inscrutable CoT look like? It would look like random tokens: “… extracurricular(boolean ger застос autism relevance directamenteдвэр QUICKchem …”
CoTs sure don’t look like that! So on a spectrum from “totally legible” to “looks like random tokens”, we are way closer to the “totally legible” side, as of today.
What does that imply about how LLMs get their superpowers? Well, imitative learning by itself will necessarily create legible CoTs; whereas in RL, the CoT is generally irrelevant to the reward, so there’s no force keeping it legible, and thus we should expect it to drift into an ever less legible and more “optimized”[2] direction.
Thus, the fact that CoTs are still pretty legible is some evidence that RLVR today has not had that much influence on LLMs, as compared to imitative learning.
1.3 LLM companies still seem to care a lot about imitative learning (pretraining & SFT) data, not just RL environments
See “Most Algorithmic Progress is Data Progress” (Beren Millidge 2025), plus further discussion in my post “The nature of LLM algorithmic progress” §1.4.
And LLM companies today continue to spend money on even more training data for imitative learning (especially expert reasoning traces for SFT, I think).[3]
1.4 Three papers claiming that non-RLVR’d models can get into the same ballpark of capabilities as RLVR’d models, although maybe we shouldn’t trust those papers too much
“Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?” (Yue et al. 2025; site, arxiv, lesswrong discussion):
The authors found that when an RLVR’d model could solve a problem, the equivalent non-RLVR’d model could generally solve the same problem, maybe not on the first try, but in one of the first 1000 tries or whatever.
“Reasoning with Sampling: Your Base Model is Smarter Than You Think” (Karan & Du 2025; site, arxiv):
The authors found that an “iterative sampling algorithm” could boost a non-RLVR’d model to be comparable to the corresponding RLVR’d model.
“Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights” (Gan & Isola 2026; site, arxiv):
The authors found that you can take a non-RLVR’d model, randomly perturb the weights, and ensemble the results by majority vote, and you can get results comparable to an RLVR’d model.
Caveat for all of these: I don’t think we should over-anchor on any of these, mainly because they’re all now outdated, and based on open models rather than bleeding-edge LLMs with bleeding-edge RLVR best practices.
Also, there’s another paper “ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models” (Liu et al. 2025, arxiv) which claims “that RL-trained models consistently outperform base models … including scenarios where base models fail entirely regardless of the number of attempts”.[4]
1.5 A paper suggesting that RLVR mostly refines the heuristics controlling which (already-known) reasoning strategy to use in which situation
“Base Models Know How to Reason, Thinking Models Learn When” (Venhoff et al. 2025; arxiv)
Using interpretability tools, the authors conclude “RL primarily teaches heuristics for orchestrating pre-existing base mechanisms, whereas SFT-distillation installs new ones”.
I think the basic idea is that, to do well in long-chain-of-thought reasoning, there are a bunch of moves that you can make: press on, backtrack, reexamine assumptions, pivot to Alternative Strategy X, etc. The LLM “knows” all those moves from imitative learning, but doesn’t have refined heuristics on which move to make in which situation. These kinds of heuristics are very important for making progress in a long CoT, and they’re somewhat domain-specific, and you can’t really pick them up well by imitative learning. So that’s the main low-hanging fruit that RLVR picks, and it makes a huge difference in practice.
Caveat: Just as in the previous section, this paper is old (by RLVR standards), and not based on bleeding-edge frontier LLMs with bleeding-edge RLVR best practices. I guess in this case I’m slightly reassured by the fact that one of the coauthors, @Neel Nanda, works with near-SOTA LLMs at his day job at DeepMind. But only slightly.
Anyway, I’m taking this claim as a good default mental picture, not an ironclad eternal truth. In particular, I remain open-minded to the Liu et al. 2025 claim that “RL can indeed discover genuinely new solution pathways entirely absent in base models, when given sufficient training time and applied to novel reasoning tasks.” (But I’m also open-minded to the opposite! Liu et al. make that claim but I don’t think they really prove it.)
(For more discussion and references on this topic, see “Some Interesting Papers on RLVR” by @CarolusRenniusVitellius.)
2. What am I actually claiming here?
What’s my point? I feel most strongly about the following intuitive picture:
We start with a randomly-initialized LLM. It “knows” nothing at all, and regardless of context, it will output random tokens like “…
OGRмагогласно વિચારDisposable! 끄registr Southeastern follic…”.Then we do imitative learning, and now we get a “base model” that (in some sense) “knows” how to speak every language in the world, and “knows” pretty much every fact that’s ever been written down, about every topic from string theory, to needle-felting, to what’s the hottest bar in Crikvenica, Croatia.[5] It has learned the human distribution.
Then we do RL, especially RLVR, where we’re mostly tweaking it to have better heuristics for which human-distribution strategies to apply in which contexts during long-chain-of-thought reasoning (cf. §1.5).
No question that the RL piece is big. It’s huge. It’s an elephant in the room. But if the RL piece is an elephant in the room, then the imitative learning piece is the friggin’ Mt. Everest in the room, if we’re trying to have a good mental picture of how LLMs can do the things that they can do.
That’s all intuition. How do we move beyond that into a concrete operationalization?
It might be interesting to look at KL divergence or weight changes between randomly-initialized models versus base models, and between base models versus RLVR? I can’t immediately find literature on that.
Or we could talk about “bits of selection”. If something like the Yue et al. 2025 results above are valid, then imitative learning might get us from
(Of course, to reiterate, all this is still compatible with RLVR being very important in a pragmatic sense: the difference between working 0.1% of the time versus 90% is a big deal!)
3. Why does any of this matter?
Here’s three debates where this topic is relevant:
3.1 Thinking about CoT legibility (both today and in the future)
There’s a nice explanation by @Rohin Shah, on the 80,000 hours podcast (recorded December 2025):
Rob Wiblin: …Just to check that I’ve understood, you’re saying because we are using GPUs and TPUs, structurally that forces the thought of the model to [not] be very deep: it can have many things in its mind at one time, but there can’t be very many steps, because you have to be going through all of these things in parallel. But if it was very [deep], then you wouldn’t be able to do them all simultaneously; you would have to wait until the earlier steps were done to do the later steps. And this is just something that is going to remain the case for years to come. …
Rohin Shah: Yeah, so that was stage number one, which is like, why should we expect the opaque serial depth to stay low, at least during the pretraining phase?
Then there’s step two, which is like, we’re saying that the pre-trained model has to essentially speak in English in order to do reasoning. But then there’s a bunch of post-training and RL and all of this sort of stuff. Maybe that’s going to make it so that even if the model is speaking in tokens, maybe it will start speaking some sort of alien language that we don’t understand.
And here I would say there’s no theoretical argument, or there’s no theorem you can prove that will say, no, they’re not going to do this. But I will say they are born speaking English in some sense, and pretraining is by far the most powerful form of getting stuff into an AI system that we have ever built. So the model is incredibly good at speaking in natural language. It’s great at doing reasoning, but only the kind of reasoning that humans do when writing stuff down.
And then when you look at the reasoning training that we’re doing today, there’s a paper from Neel Nanda and others that shows that actually a substantial part of what the reasoning training is doing is just teaching the model when to do a specific kind of reasoning step that it had already learned during pretraining.
So basically a lot of the capability is just coming in pretraining. We know that the pretraining, for good reason, is going to stay the way that it is, and be done using human-like reasoning and speaking in English. And then the RL is just really inefficient relative to pretraining, and for it to build an entirely new epistemic language that is something that we wouldn’t be able to understand [even] with a decent chunk of effort, is just so far beyond what RL is doing currently that it would be pretty surprising to see that in the near future.
Now, do I expect to see it ever? Yes. But I think six months ago, I split a room in a conference by saying, “Agree or disagree? Chain of thought monitoring will continue for two years.” I think I said two years. It might have even been one year. I forget. Whereas my median is probably like four years, five years, I don’t know. It’s not totally clear, mostly because of this argument about how hard it would be to go beyond the current situation.
This is kinda the mirror image of my §1.2 argument above. I was saying in §1.2: CoTs are legible, therefore LLM capabilities mostly come from imitative learning. Whereas Rohin is saying: LLM capabilities mostly come from imitative learning, therefore CoTs are legible, and this will not change too soon, absent some important future change in LLM training approach.
3.2 Thinking about LLM capabilities (both today and in the future)
Insofar as LLMs are powered by imitative learning, we should expect them to mainly be competent in domains where there is human data and human concepts to learn from.
Insofar as LLMs are powered by RLVR, we should expect them to mainly be competent in domains where there are verifiable (automatically-checkable) rewards.
I think both are relevant. And indeed, LLMs seem to be at their best when we have both. For example, software debugging allows both copying from abundant human concepts and human reasoning traces, and testing results in a controlled environment.
By contrast, I have heard lots of people talk as if verifiable rewards are sufficient by themselves—that as long as we have verifiable rewards for such-and-such domain, LLMs will sooner or later be able to master that domain, even if the domain is arbitrarily far from anything humans have ever thought about. Well, that’s a hypothesis. Is it true? Seems to me that the jury is out; I don’t recall seeing demonstrations of such a thing to date. Anyway, I think my headline claim (“LLMs are (still) mostly powered by imitative learning, not RL”) provides some helpful background intuitions and context as we try to think through what to expect here.
3.3 Thinking about LLM alignment (both today and in the future)
I think a very strong general rule-of-thumb is: if you create AI capabilities via RL,[7] you get ruthless sociopathic behavior. (See my post “Why we should expect ruthless sociopath ASI”.) The logic is basically: whatever the reward function is, reinforcement learning is reinforcing the ruthless and relentless pursuit of a high score on that function, by any available means.
By contrast, if you create AI capabilities via imitative learning, you get models that follow the human distribution of outputs. (Cf. “personas”, “simulators”, etc.) The human distribution has its problems, Lord knows, but it’s not nearly as cursed as the “100% ruthless sociopath all the time” that you normally get from building capabilities via pure RL training.
For example, the Bing-Sidney chatbot from 2023 was trained by pure imitative learning (pretraining + SFT), without any RL at all.[8] Was it misaligned? Hoo boy, was it ever! But it had a very human flavor of craziness, in contrast to the very RL-ish flavor of ruthless-goal-seeking craziness that we often see these days (and this week in particular!).
Anyway, that’s my proposal, and I claim that it’s fully compatible with the alignment-related evidence that we’ve seen from LLMs to date. They are definitely not 100% ruthless sociopaths all the time. But they are somewhat like that sometimes. Correspondingly, RLVR is part of, but not most of, how LLMs get their capabilities today.
If we want to refine that idea, we get into my “Foom & Doom” §2.3.5 picture where LLMs have some niceness from imitative learning, and then the more you do RLVR, the more that niceness gets diluted away.
Copied from “Foom & Doom” §2.3.5
There seem to be ways to mitigate the damage of RLVR (e.g. inoculation prompting), but that’s still directionally what I strongly expect to happen.
How fast does RLVR dilute away the niceness? Unclear. But if LLMs are (still) mostly powered by imitative learning, not RLVR, and yet this (comparatively) small amount of RLVR is already enough to cause obvious RL-style alignment problems, then that’s a bad sign for our ever-more-RLVR LLM future.
- ^
There are a couple edge-cases in this breakdown, and the way we should handle those cases depends on what we’re ultimately trying to figure out. My interests are listed in §3, and in those contexts I would say: (1) I agree with Arthur Conmy’s take that Direct Preference Optimization (DPO) usually has more in common with imitative learning than RL, even though it’s sometimes lumped in with RLHF; (2) If Company A uses RLVR, and the CoTs get posted on the internet, and then Company B does SFT on those CoTs, then this should count as RL even though it’s structurally imitative learning. These edge-cases won’t really come up anyway; the paradigm case that I want to talk about is a big AI company doing pretraining on human-created internet text, SFT on their own proprietary data, and RLVR on their own proprietary training environments.
- ^
Cf. the Karpathy quote: “You can tell the RL is done properly when the models cease to speak English in their chain of thought”.
- ^
How much money? Not sure. The total amount that LLM companies are spending on “data” is huge (e.g. “$10 to $15 billion per [LLM company]” per year), but that claim seems to involve a broad definition of the word “data”, that combines both imitative learning data (pretraining & SFT) and RL training environments. I can’t easily figure out the breakdown in spending between imitative learning versus RL. However, the imitative learning part definitely seems to be more than zero. E.g. a recent blog post by an industry insider mentions “high-quality midtraining tokens”.
- ^
The paper says “regardless of the number of attempts”, but that raises the obvious question “well, how many attempts did you try?” The paper does not make that clear, unless I missed it. Their graphs top out at pass@256 (i.e., probability of succeeding at least once, given 256 tries), but they must have done a bunch of pass@256 runs to get a pass@256 probability. Based on the plots, maybe
- ^
(A town I picked randomly by pointing at a world map.) Gemini 3.1 Pro without internet access suggests Sabbia. I’d go!
- ^
As mentioned above, Liu et al. 2025 found tasks where RLVR succeeded but the base model pass rate was below
- ^
Note that RLHF is not really a “creating AI capabilities”; rather, conventional wisdom is that RLHF makes models dumber, not smarter, as a side-effect of making them nicer. Again, this post is mostly about RLVR and similar.
- ^
Source: see this comment by @gwern.
FYI when I was looking into this the AI I was asking about it guessed more like 20% a couple months ago. This was based on public models and some reading between the lines about private frontier models.
I don’t strongly believe 20% is more accurate than 50% but curious if you remember more details about where the 50% number came from.
I forget. Definitely not a reliable source. In fact, I’ll edit the guess to 20% now. If anyone knows more, please share.
Dumb question but if RL is so much less compute efficient than imitation learning, why are the companies spending 20%+ of their compute on it? Under your model, I’m not surprised they’d use some RL, the first doubling of a new technique is the cheapest etc, but it does seem surprising/something needs to be explained for a 80-20 split.
I’d roughly expect the AI companies to roughly spread their compute usage out until marginal gain from any given method equals any other. So if RL is much less scalable/compute-efficient compared to imitation learning, you should be surprised that the 21st percent of compute used on RL is similarly effective to the 81st percent percent of compute in Imitation Learning, no?
Curious where my model here breaks down.
Hmm, I reworded the section heading
FROM “Theoretically, each GPU-hour spent on RL should have orders of magnitude less contribution to LLM capabilities than a GPU-hour spent on imitative learning”
TO “Theoretically, each GPU-hour spent on RL conveys orders of magnitude less information content than a GPU-hour spent on imitative learning”
Sorry about that.
The RLVR is a small change to the model in the grand scheme of things—that’s my point—but of course it’s a small change that makes the model much better at things that the companies care about. If there was a way to get that same small change via imitation learning, then that would require much less compute, and companies would definitely want to do that. But no such alternative is known. …Well, oh actually, there is a way to do that: you can find some other model that can already do good inference-time reasoning and then distill it (via imitation learning). And companies do exactly that whenever they can. But you can’t push the SOTA that way. So they pay the cost.
My quick guess-answer is that “compute efficient” here is abstracting away the fact that pre-training trains capabilities we don’t take care about, while RL trains capabilities we care about. For example, if only 1 in 10^3 of the “bits of learning” pretraining does go towards bits we care about, then RL, being much more targeted to stuff we care about, might get a free (up to) 10^3 efficiency increase.
Curious to hear your thoughts.
I agree that the information in terms of bits that get into the LLM almost all come from pretraining. But certainly, some bits are far more important than others.
This is a simplification. But you could imagine all the facts “Paris is in France”, “Baker Brun is known for its skillingsbolle”, are put into it by pretraining, along with everything else.
Then you can imagine that RL mostly modifies just the “higher level” things. Like the in-context learning algorithms the model has, its “general reasoning” abilities, its “goals” and “dispositions”:
Seems like these higher level things should have very little information content — I can write a PPO+adam+lora+weight_decay+entropy_regularization+kldivergence script in 500 python lines.
But they should be of massively outsided importance. Whether the model knows which pastry Baker Brun is known for would seem to me to be of ~0 relevance to alignment. Its goals, reasoning abilities and in-context learning abilities strike me as where ~100% of the meat is.
This makes me think arguments like
LLMs RL stage inputs very little info
The parameter update caused by RL is
small
has low rank
The KL divergence between an RL’d model and a non-RL’d model is small
Tell us very little about the impact of RL on the alignment or danger profile of models.
——————————————
This still leaves us with this (very good) picture you’ve drawn. Its just, I don’t think we have any reasonable bound on how quickly SFT+RLHF’s niceness gets dilluted away by RL.
(edited 7 minutes later, to delete a part of my comment because I had misunderstood what you wrote.)
In the big picture, I basically agree with all that. But I’ll nitpick a bit anyway :)
I agree that “knowledge” of Baker Brun in particular is not related to alignment. But we shouldn’t generalize from that example to saying that pretraining is irrelevant to goals. (Maybe you didn’t mean to insinuate that anyway?)
For example, a base model may well autocomplete “I’m cold” to “I’m cold, so I’m gonna put on my coat now!”, which is pretty goal-like. Granted, it’s still not a true goal yet, but once we bring in tool use, those same autocomplete expectations can turn into bona fide goal-seeking actions. And yet they’re still derived from pretraining, and still reflective of the human distribution.
I don’t know how to bound it from first principles, but at least we have some empirical data by now.
I guess just, I don’t quite understand what “LLMs are primarily driven by imitation learning” gets us.
If there was a way to show that “99% of LLM goals and capabilities come from pretraining” (however ill-specified that), I’d be somewhat comforted.
But seems to me the statement we can actually be confident in is more like “LLMs get their base ontology mostly from pretraining, and get a bunch of cursed entangled correlations from pretraining, and then RL shapes it to varying degrees.”
And it could be we end up with a crisp ruthless sociopath with some residual reflexes/habits from pretraining. Or a really gung-ho human with some ruthless efficiency-maxxing instincts they don’t endorse on reflection. Or a
p̟͓͔̈́͆͋͝a̵̠͕̟̔͑ṕ̵͔̀̉̄̚e̷͕͒̓͐͘ṟ̴̊͒̎͆̚͠c͖̮̉̾̊͊̔l̴͙͈͔̣͒̍̈̅͆i̸̠̜̿͂̉͂͋͠p̰̤̐̓̈̃̏͐p̵̛̱̟͈̔̾̕ę͕̫̤̔̈́̓̚͠ṟ̹̩̯͇̲̑̀͗̓͗.Like, what we care about are a series of qualitative properties the models might and might not have, and they’re not at all reducible to qualitative metrics like the ones above.
———
Another intuition I have is just, the pretraining distribution is really wide. There are sociopath notebooks somewhere, people who’ve translated documents into Ithkuil or Lojban, there’s like proof passages that are +3 SD levels of brilliance from the distribution of proof passages generated by humans +5SD of brilliance. And there’s stories about aliens deliberately written to be very strange in various ways.
If you kind of dynamically pastiche these together and amp them up and distort them in various ways, in service of pure reward maximization. I can’t really tell you what you get. Which makes me less comforted by a pretrianing anchor.
Yeah I think that’s overall a very reasonable stance on LLM alignment.
I recently wrote Reinforcement learning scaling might incentivise hidden reasoning architectures for AI. It’s perhaps taking similar lines of reasoning but (tentatively) interpreting some of the claims and observations differently.
Vika and others (you mentioned Rohin) seem to both be sort of a bit worried that RL or other developments might kill the fragile opportunity afforded by legible CoT.
I take you to be (among other things) suggesting this isn’t substantially a concern. I’m quite unsure!
One point I wonder how you’d respond to: it matters here what the compute expense of each stage is, as well as its relative contribution. This is because the compute expense shapes the incentives and constraints on architecture choices. Pretraining-cost-dominated pipelines strongly incentivise recurrence-limited (e.g. transformer) architectures, which almost by (accidental) design enforce a certain amount of externalised reasoning and therefore (fallible, but real) CoT legibility. As that pretraining dominance in cost dwindles, that architectural incentive diminishes. This is independent of the relative contribution of each stage to the actual competences of the resulting model.
You might say: the competences are one thing, but the inclinations/drives are another. That’s where the really important questions are, and those are questions of contribution, not cost? Though it seems like (regardless of the information-theoretic or sample-effective questions of contribution) RL often makes them ruthless!
I edited a sentence in OP: It used to say “…Whereas Rohin is saying: LLM capabilities mostly come from imitative learning, therefore CoTs are legible, and this will not change too soon”, but now it says “…this will not change too soon, absent some important future change in LLM training approach.” I agree that this is an important caveat, thanks.
I have no opinion about whether there will be important future changes in LLM training approach. What you said sounds like a plausible consideration, sure, but I dunno.
“CoT monitorability is a fragile opportunity” seems like a fine framing to me. I mean, we can pessimistically emphasize how CoTs are not a certain panacea for safety, or alternatively we can optimistically emphasize how CoTs are not always completely useless for safety. But that’s just a vibes disagreement, because both are true.
Yeah, for what it’s worth, it’s disappointing to me how heavily interpretability-loaded (CoT included) many leading safety plans are, and I’m cynical about that for other reasons (like “people just won’t really bother looking at the CoT much anyway”, gestured in the conclusion of that post).
Thanks.
Does this change your fundamental views on risks from LLM based AI systems?
Risks depend on both capabilities (could it do Bad Thing X if it wanted to?) and alignment (does it want to?).
LLM alignment: The §3.3 discussion is my take on that, and it hasn’t changed for a long time (e.g. compare with Foom & Doom §2.3 from June 2025, and the non-RLVR part of that is in turn parroting §4.2 of this post I wrote 2023 …).
LLM capabilities: I didn’t discuss this in the OP, but my opinion is still that there’s a certain kind of “figuring things out” that humans can do (especially over extended periods of time), but that LLMs can’t, not now and not ever. (I’m stating an opinion without defending it.) But it’s tricky for me to translate that hypothesized limitation into concrete predictions of what future LLMs will or won’t be able to do in the real world. Could future LLMs wipe out humans, invent science and tech centuries beyond our wildest imagination, and colonize the galaxy? I’m confident in “no”. Could LLMs wipe out humans, leaving aside the question of whether they’d be able to survive on their own afterwards? I still lean “no”, but less confident. I’m certainly happy for there to be people working on LLM x-risk, and indeed I think there should be way more work going into that. But I also think the non-LLM thing that I’m working on (“brain-like-AGI safety”) is an EVEN scarier, more neglected, and more likely x-risk on the horizon.
I think “it would also be 0.01% of changes to the weights” is not in fact entailed? There is a zeroth order baseline that it’s more like 1:1, because when you get a successful transcript on a problem in RL, roughly speaking you just as-if-pretrain on that transcript (like, you get a gradient contribution from asking for an increase in the probability of each token of that transcript). That said, I think there are important corrections away from this.
edited to add: oops ok i see i was probably misunderstanding you: i was talking about the per token learn rate in each case; you were talking about the per RL transcript learn rate vs the per pretrain token learn rate. i would guess that my normalization is somewhat more natural as a zeroth order baseline, but not sure and mostly nvm the point above then, sorry. see item 3 of the list below
I will now list some considerations on the general topic of RL vs pretrain thinkoomph contributions off the top of my head, without putting in the work to put these together into some coherent story atm:
1. A model-generated transcript that solves a problem will be close to what the model already does naturally, so there is less for the model to learn from it — like, a human’s transcript on the same problem would exhibit “tricks” that are more novel to the model.
2. otoh: A model-generated transcript will be natural for the model, so the tricks in it are easier for the model to learn — like, a human-generated transcript for the same problem would be more alien for the model; each trick exhibited in it is harder for the model to learn/use.
To illustrate the above two points with a hyperbolic example: it’s somewhat like a teacher guiding your thinking a bit when you’re trying to solve a problem vs telling you a solution in chinese and trying to get you to repeat it back.
3. Fable’s guesses are that for frontier models, pretrain token vs RL token weight delta sizes are 3x in favor or pretrain, whereas pretrain token vs RL transcript weight delta sizes are 300x in favor of RL, with wide uncertainties.
4. Under “RL”, I think labs have in part been doing some things other than vanilla RL, and I think this could easily contribute a >10x factor to thinkoomph calculations. Some examples in this category (various of these are kinda iterated amplification and distillation):
4.1. Have a model try to solve a problem. Have a human prompt the model with corrections and other hints until it solves the problem correctly. Then train on the transcript with these hints removed. Or maybe train on the transcript with these hints, idk. (I think labs have effectively (indirectly) been employing at least on the order of 100k graduate-student-likes to provide training feedback, including this sort of thing as a significant part.)
4.2. Have the model distill a long solution path to a streamlined one that eg cuts out bad branches, then train on that streamlined one.
4.3. Have the model spend a lot of time solving a problem, then write hints, then prompt a model to solve the problem with these hints, then train on the transcript with the hints removed.
4.4. The previous item but with math textbooks or paper fragments etc in context.
4.5. Give the model the/an answer to the problem, ask it to write a way to get to that answer, then train on this rationalization.
fwiw my own high-level guess is that the understanding/skills of current models are in some sense (that is hard to make precise) more from humans than originally created, but my guess is that at this point it’s significantly less extreme than mt everest vs an elephant