The first counterargument is that LLMs and humans learn in different ways. For a human, creating a new language is an independent, strenuous task. For an LLM, every single token embedding is being constantly updated as a side-effect of whatever else it’s doing. It is constantly “making up new languages to solve a problem”, it’s just that most of these problems, like emulating text in an SFT dataset or maximizing human feedback rewards, involve sounding a lot like a human does.
The second counterargument is that languages—even in humans—do not suddenly split off and form new ones that have to compete with each other. They drift over time. Indeed, we see exactly this in the DeepSeek paper—the authors alternate between RLVR optimization and SFT to prevent a drift away from comprehensible English[1].
For an LLM, every single token embedding is being constantly updated as a side-effect of whatever else it’s doing
This is also true of humans, albeit we cannot locate the embeddings as easily. Every single thought you have is updating your embeddings, and giving you a different internal language—see Quine’s analogy of the bush for language.
Indeed, we see exactly this in the DeepSeek paper—the authors alternate between RLVR optimization and SFT to prevent a drift away from comprehensible English[1].
You’ll note that the penalty in the DS paper to unintelligible language is compatible with both “sampling from a less intelligible distribution of midtraining / pretraining” and “inventing a new language.” I.e., even if DS were entirely recapitulating word-for-word someone’s chain-of-thought solving a math problem, if someone writing this math problem was using a shorthand the penalty for non-English text would still get triggered and we’d still see a reduction in performance subsequent to requiring completely intelligible text.
The first counterargument is that LLMs and humans learn in different ways. For a human, creating a new language is an independent, strenuous task. For an LLM, every single token embedding is being constantly updated as a side-effect of whatever else it’s doing. It is constantly “making up new languages to solve a problem”, it’s just that most of these problems, like emulating text in an SFT dataset or maximizing human feedback rewards, involve sounding a lot like a human does.
The second counterargument is that languages—even in humans—do not suddenly split off and form new ones that have to compete with each other. They drift over time. Indeed, we see exactly this in the DeepSeek paper—the authors alternate between RLVR optimization and SFT to prevent a drift away from comprehensible English[1].
I have seen the same in other papers, though they’re sufficiently niche that it’d be personally identifiable to name them.
This is also true of humans, albeit we cannot locate the embeddings as easily. Every single thought you have is updating your embeddings, and giving you a different internal language—see Quine’s analogy of the bush for language.
You’ll note that the penalty in the DS paper to unintelligible language is compatible with both “sampling from a less intelligible distribution of midtraining / pretraining” and “inventing a new language.” I.e., even if DS were entirely recapitulating word-for-word someone’s chain-of-thought solving a math problem, if someone writing this math problem was using a shorthand the penalty for non-English text would still get triggered and we’d still see a reduction in performance subsequent to requiring completely intelligible text.