People sometimes make bit-counting arguments about RL, usually to say something about capabilities or alignment, something like: A million pretraining tokens puts several million bits of selection pressure on the models weights, but a million token RL rollout only puts a single bit of selection pressure.
I think people sometimes forget that with betteralgorithms you can get much more than one bit per RL rollout, though not as many bits as in pretraining. This is not a new idea, e.g. even Ilya’s 2023 paper on process supervision provides ~20 bits/rollout. IMO part of the reason that we’re getting more neuralese-like CoTs is that people are using more efficient RL algorithms (as well as just scaling up RL).
I think people sometimes forget that with better algorithms you can get much more than one bit per RL rollout, though not as many bits as in pretraining. This is not a new idea, e.g. even Ilya’s 2023 paper on process supervision provides ~20 bits/rollout. IMO part of the reason that we’re getting more neuralese-like CoTs is that people are using more efficient RL algorithms (as well as just scaling up RL).