David Oks argues that LLMs are weird (e.g. the goblins) because you can’t have good adaptive learning in environments with coarse and sparse feedback without also overindexing on the weird tics; minimising costly false negatives essentially leads to high false positives, so LLMs cargo-cult. RLHF causes this, RLAIF accelerates this, maybe mature mechinterp might fix this.
So where did the goblins come from?
If you were to answer in one word, you could do pretty well with “overfitting.” If you were allowed to answer in two words, you could perhaps do a better job with “chunky post-training.” … The models were trained to do something; the feedback signal from that training process was coarse enough that the model struggled to distinguish which features of its output earned the reward and which ones were incidental; and the result was that along with the things it was trained to do, the model picked up weird quirks of its learning regimen. And then those quirks were magnified with each generation of model, since the outputs the models produced were used to train future generations of models. …
There are two ways to understand these quirks. The first is the natural engineering response: these are technical bugs, artifacts of imperfect training, and they reflect how different language models are from human brains. I don’t think that this view is necessarily wrong. But [the other way] is that large language models are adaptive systems: they are systems that adjust their behavior in response to feedback from the outside world, and make that adjustment in a way that tends to improve future performance.
But a lot of the time, getting feedback from your environment is hard. … Adaptive systems of all kinds frequently face situations where feedback is both coarse and sparse: that is to say, feedback is infrequent and in small quantities, and it arrives in a way that makes it hard to decipher what exactly worked and what didn’t. In these circumstances, where the causal structure of feedback is opaque, it’s hard to know what input caused what output: and the proper response, in these conditions, is to overindex and overimitate, to learn too much—since learning too much from what worked is better than learning too little. This is true for all adaptive systems that encounter sparse and coarse feedback from their environment. And the inevitable result of their overlearning is that useful traits are bundled with all sorts of inert, eccentric, or even harmful ones: in other words, that adaptive systems are weird. The reason they’re weird is the same reason they work at all.
Consider, for example, the wonderfully powerful and strange adaptive system that we call human culture. In 2015, the anthropologist Joseph Henrich published a book called The Secret of Our Success. In the book, Henrich was trying to answer a particular question. The individual human, he pointed out, is not very impressive. We aren’t particularly fast, or particularly strong; our problem-solving abilities are impressive, but aren’t orders of magnitude above those of other primates. … So why did humans win? Henrich’s argument was this. What ultimately set humans apart wasn’t our individual cognition but our unusual capacity for high-fidelity social learning: that is to say, our capacity to develop culture, the thick accreted body of practices, tools, skills, beliefs, and customs that define human life. … If feedback cycles are long and the feedback itself is coarse, then it’s hard to know why someone succeeded. The nixtamalization process, for example, was bundled with practices that didn’t do anything in particular, like blowing on the maize before putting it to cook or swaddling certain cobs like newborns and letting them sit outside the house all year. But if all you know is that the process as a whole seems to prevent pellagra, then the optimal thing is to imitate the entire bundle of practices. … Why is that? For simple reasons of cost. If you copy a few useless steps in a process, the cost to you is relatively low; but if you omit the single step that actually matters, you could die. And so Henrich says that human culture is shaped not just by imitation but indeed by overimitation: and that overimitation is the source of all sorts of weirdness within every culture. One can imagine that with enough time, these maladaptive practices would be selected out via the same process of cultural evolution that created them. [But] we simply haven’t had enough time to converge on “optimally adaptive” cultural organisms. There have been about 1,500 human generations since the first cave drawings, and perhaps 12,000 since Homo sapiens first emerged; that is a very short time period for a coarse feedback mechanism to do its pruning.
Now let’s consider another adaptive system, of a very different kind: the large language model. … The pretraining process is extraordinarily dense, with tens of trillions of microcorrections localized to specific tokens; but post-training is quite different. The SFT and RL stages that characterize post-training involve orders of magnitude fewer training events; and because they score entire outputs rather than specific tokens, the feedback the models receive is much more coarse than what they receive from pretraining. … In this way pretraining is akin to biological evolution, and post-training to cultural evolution. And the result is what we’d expect from the logic of adaptive systems: post-training frequently produces eccentricities of all kinds in language models. The catchiest term I’ve heard for this phenomenon is “chunky post-training.” Post-training is composed of discrete “chunks,” and from these chunks the model encodes “unintended correlations”: “correlations between formatting and content, narrow phrasings across diverse problems, and implicit associations arising from the discrete data curation process.”
Is there a solution? At the limit probably not: there will always be some residual strangeness in these systems. But we can certainly reduce it by making the world’s feedback less coarse and less sparse, and by trying to understand why things happen—why feedback goes one way and not another. Our best tool for this, as humans, is scientific investigation, as expressed through the mechanisms of controlled experimentation, variable isolation, and formal causal inference… We’re still in the early stages of defining the equivalent science of causal structure for language models… mechanistic interpretability, which seeks to understand the internal computations and states of a neural network, has made huge progress in the last few years, and it could be revolutionized in the years to come by automated interpretability researchers. But as a science it’s still in its infancy, sitting somewhere between classical psychology and early neuroscience. …
David Oks argues that LLMs are weird (e.g. the goblins) because you can’t have good adaptive learning in environments with coarse and sparse feedback without also overindexing on the weird tics; minimising costly false negatives essentially leads to high false positives, so LLMs cargo-cult. RLHF causes this, RLAIF accelerates this, maybe mature mechinterp might fix this.