RL is much less efficient per token: one scalar reward every several thousand tokens instead of a label per token. But the per-token value is different to the marginal value of each unit of compute invested which is what determines compute allocation. The 80⁄20 split tells you the two are roughly equal in value at the margin.
What makes them equal at the 80⁄20 equilibrium is that pretraining has diminishing returns. For pretraining the relationship between loss and compute is a power law (roughly C^-0.05, so 10x compute reduces the remaining reducible loss by ~11%) and given the size of pretraining runs now, the marginal value of pretraining is low enough to justify allocating 20% of compute to RL.
Also it depends on how you measure efficiency. Pretraining wins on efficiency per token but RL is often better at efficiency per unit of human effort. For example, writing a verifier function for a math problem takes much less human effort than writing out all the steps in the solution.
RL is much less efficient per token: one scalar reward every several thousand tokens instead of a label per token. But the per-token value is different to the marginal value of each unit of compute invested which is what determines compute allocation. The 80⁄20 split tells you the two are roughly equal in value at the margin.
What makes them equal at the 80⁄20 equilibrium is that pretraining has diminishing returns. For pretraining the relationship between loss and compute is a power law (roughly C^-0.05, so 10x compute reduces the remaining reducible loss by ~11%) and given the size of pretraining runs now, the marginal value of pretraining is low enough to justify allocating 20% of compute to RL.
Also it depends on how you measure efficiency. Pretraining wins on efficiency per token but RL is often better at efficiency per unit of human effort. For example, writing a verifier function for a math problem takes much less human effort than writing out all the steps in the solution.