Well, if RL “really worked”, yes it seems like it should almost certainly invent and reason in an alien language. But with all the models where we can actually see the CoT, it’s more or less in English. My interpretation (and I think the standard interpretation?) for that is that RL is hard and can’t really optimize too far from the parameters found in pretraining.
That step isn’t necessary for the argument. The point is just that you’re encouraging extra compression rather than incentivising the model to invent a language from scratch.
Well, if RL “really worked”, yes it seems like it should almost certainly invent and reason in an alien language.
What would be the advantage? Inventing new languages at least doesn’t seem to increase our reasoning ability compared to using existing ones. For example, it seems unlikely that people can reason better with any artificial languages like Esperanto or Interlingua (or Python, UML, etc) than with English. I would expect that if it were possible to artificially create a language that is substantially better for thinking than existing languages, we would have already created such a language.
Of course humans can also reason to a significant degree in latent space, without language, but latent reasoning (Neuralese) is different from reasoning in an artificial language (Thinkish).
I’m not sure whether I’d expect a whole new language with its own grammar. But with sufficiently capable RL, I do expect new words, as handles for new concepts the model came up with.
I guess the general argument is that the optimization space is very high-dimensional, and there’s no pressure to reason in English, so it just seems incredibly unlikely that the optimum would look like thinking aloud in English. I’m not sure how much I buy the human analogy; we have different capabilities and IMO humans don’t think in language to the same degree that current LLMs do. But certainly humans in specialized areas find it useful to invent lots of specialized terminology and conventions. If you sent people to an island and had them live for 10000 years with their society entirely devoted to chess, I’d conjecture that their language would evolve quite a bit. If RL could “really do RL” it should be able to create such evolutions for many domains, or (I suspect) cross-domain.
This seems out of line with what was said in the GRPO paper way back when. They had to alternate between RLVR and supervised learning to make sure the model still spoke English properly when it was done.
DeepSeek (which GRPO was invented for) still uses a normal English CoT to this day. (V4 Pro) They had to alternate with supervised learning because RL is just very very hard and GRPO is incredibly weak compared to supervised learning. They aren’t deliberately sacrificing performance by forcing an English CoT instead of a more useful or efficient neuralese. They are forced to do it because RL just doesn’t work that well.
“On the logic that reinforcement learning can accomplish more than was understood.”
Could you explain this? It seemed obvious that RL results in difficult to parse CoT.
Well, if RL “really worked”, yes it seems like it should almost certainly invent and reason in an alien language. But with all the models where we can actually see the CoT, it’s more or less in English. My interpretation (and I think the standard interpretation?) for that is that RL is hard and can’t really optimize too far from the parameters found in pretraining.
“invent and reason in an alien language”
That step isn’t necessary for the argument. The point is just that you’re encouraging extra compression rather than incentivising the model to invent a language from scratch.
What would be the advantage? Inventing new languages at least doesn’t seem to increase our reasoning ability compared to using existing ones. For example, it seems unlikely that people can reason better with any artificial languages like Esperanto or Interlingua (or Python, UML, etc) than with English. I would expect that if it were possible to artificially create a language that is substantially better for thinking than existing languages, we would have already created such a language.
Of course humans can also reason to a significant degree in latent space, without language, but latent reasoning (Neuralese) is different from reasoning in an artificial language (Thinkish).
I’m not sure whether I’d expect a whole new language with its own grammar. But with sufficiently capable RL, I do expect new words, as handles for new concepts the model came up with.
I guess the general argument is that the optimization space is very high-dimensional, and there’s no pressure to reason in English, so it just seems incredibly unlikely that the optimum would look like thinking aloud in English. I’m not sure how much I buy the human analogy; we have different capabilities and IMO humans don’t think in language to the same degree that current LLMs do. But certainly humans in specialized areas find it useful to invent lots of specialized terminology and conventions. If you sent people to an island and had them live for 10000 years with their society entirely devoted to chess, I’d conjecture that their language would evolve quite a bit. If RL could “really do RL” it should be able to create such evolutions for many domains, or (I suspect) cross-domain.
This seems out of line with what was said in the GRPO paper way back when. They had to alternate between RLVR and supervised learning to make sure the model still spoke English properly when it was done.
DeepSeek (which GRPO was invented for) still uses a normal English CoT to this day. (V4 Pro) They had to alternate with supervised learning because RL is just very very hard and GRPO is incredibly weak compared to supervised learning. They aren’t deliberately sacrificing performance by forcing an English CoT instead of a more useful or efficient neuralese. They are forced to do it because RL just doesn’t work that well.