Unblocking AI’s Continual Learning: Hints From How Humans Learn
If you’ve ever screamed in all-caps at an AI, then you know the difference between what it learned when it was trained, and what you can teach it by prompting. The LLMs powering today’s AI don’t learn on the job the way people do. They learn all at once in a big training run and once that’s done, we freeze the parameters that store their skills and knowledge.
So we all get the same AI with the same skills and biases, centrally trained by a frontier model company. Beyond turning everything even more same-y, there’s an economic cost to this centralization: firms use AI that lacks understanding of their unique rules, culture, and quirks. Humans learn this “tacit knowledge” on the job through observation and (sometimes painful) feedback, but AI with its frozen parameters cannot. With context engineering, we can augment the prompt to help AI remember facts, but not teach it skills that last.
It’s not surprising that the domains where AI is most successful, like coding, are those suited to centralized training. Good software development skills are mostly firm-agnostic. For everything else firm-specific, AI is trained to trawl the codebase and build context from scratch for every single task. Human developers don’t do this. It would be absurd to block off a week for “establishing context” before starting every task. Human developers develop durable conceptual models of the codebase over time. From-scratch context is a band-aid that works for AI coding because codebases are a good source for up-to-date context (with some exceptions). Since AI doesn’t learn on the job (a capability called continual learning), we need an analog for the “codebase” before we can replicate AI’s success with coding in other domains. But no such analog exists universally.
In 1945, the economist Friedrich Hayek described what became known as “Hayek’s knowledge problem”, which states that critical economic information needed to run firms efficiently is dispersed and local, and not available centrally. To this day, it remains a core argument in favor of markets, and against central economic planning. So it’s wild that some mainstream economists now argue that AI makes central economic planning more likely[1], in part by making tacit knowledge more codifiable. This twist aside, more codified tacit knowledge doesn’t solve Hayek’s problem because without continual learning, AI won’t be up to date with it.
For all of these reasons, AI that continually learns tacit knowledge will be far more economically valuable than today’s AI. But the effect won’t be uniform for every application of AI. One of the most promising and popular such applications, (AI remains otherwise deeply unpopular) is accelerating science for the discovery of new therapies and materials. For that, continual learning isn’t optional. No training data (human-generated or synthetic) covers what’s off the edge of the scientific map. Hundreds of millions of dollars are flowing into companies that are building scientific loops around AI, but unless it’s able to continually learn from experiments like human scientists, how far could it really go beyond what’s already known?
Xerox Builds an Expert System AI
In contrast to today’s AI, humans learn continually but it takes time. For example, when moving between firms within their own discipline, humans take years to get back to their original productivity.[2][3] So companies have an incentive to find ways of accelerating the process. One such company was Xerox. In the early 1980s, it was struggling to recruit enough ex-military technicians to service its machines in the wild, and wanted to lower the skill bar for new technicians.
The result was documentation in the form of decision trees that guided a technician from symptoms to diagnosis. The trees were built by sampling and diagnosing faults in the lab. A decade later, AI researchers building expert systems at Xerox’s PARC sought to use their models to replace the decision trees.[4] They built a system called RAPPER for a single module of the machines, and demoed it to the technicians. When asked whether it would be useful if scaled up to cover the whole machine, this is what they said:
“Not really – though it is amazing – rather like a bear dancing.”
RAPPER was based on an engineering model of the module and did not learn over time from the technician’s experience. RAPPER was faithful to the decision trees in the documentation (it covered every fault they did), but the hard problems were new ones not in the documentation. Looking for insights, researchers rode along with the technicians on service jobs. What they found were heuristics, stories swapped at work-group meetings, and a radio network which technicians used to solicit ideas from buddies if they were stumped. They got an idea:
It suggested to us that we could stand the artificial intelligence approach on its head, so to speak; the work community itself could become the expert system
RAPPER couldn’t learn over time, but the researchers could build a tool to make it easier for the technicians to learn from each other. The tip economy of the technicians became the blueprint for the system that eventually saw adoption: Eureka was not an AI, but rather a searchable database of validated tips collected by the field engineers and technicians. These tips were the definition of codified, tacit knowledge: technical and specialized to Xerox. For example:
Diagnosing unusual, costly failures—Bimetallic corrosion builds up on A and causes intermittent failures that seem to be B. Replacing B makes the problem seem to go away because A is moved in installation. First clean A, and later replace by new gold-plated AA, available as Part #1234...
Eureka added value to Xerox because the hard problems were new and few technicians knew how to fix them. In such a data-poor environment, the technicians could learn from each other with Eureka, rather than from just their own direct experiences.
Tacit Knowledge Has Few Samples, and LLMs Need Many
Xerox is not unique in needing employees to learn specialized tacit knowledge to contribute. Such knowledge is specific to a firm or even team, proprietary, and mostly not written down. Some of it, like the knowledge the Xerox technicians learned about diagnosing and fixing machines, can’t easily be written down at all. It needs to be learned from a few examples, or “samples”.
Humans (and animals) are good at learning from a few samples. There’s likely evolutionary selection at play, especially for learning harmful associations. For example, rats that are given sweet water then injected with a drug that makes them sick learn negative associations from only a single experience,[5] which makes sense: If you eat a berry and get sick, it’s useful to your survival to never eat another one. Even if you got sick from something else, it’s better to miss out on a few tasty berries than eat a poisoned one.
The LLMs powering AI in comparison are not sample-efficient during training, lagging somewhere between 3 and 6 orders of magnitude behind humans,[6][7] which is a problem if you want them to learn a firm’s tacit knowledge from a handful of samples in the wild.
But wait, can’t we teach LLMs by adding a few examples of what we want to the prompt? Yes, however, there’s no way to commit what the LLM learns from the prompt samples (a process called “in-context learning”) to the LLM’s parameters. So the examples stay in the prompt. But the prompt has finite length. If we naively push new observations into it, old ones will have to drop off, and we’ll lose what’s learned from them. Maybe we could solve the length issue by curating just the right samples into the prompt for each task? We’d store them in a database, then grow that with new observations. The prompt stays open for just the samples relevant to the current task. But observations will keep coming in, and eventually the total samples relevant to a task won’t fit again. We’ll need a way to “compress” the samples into an abstract form. And the abstraction should generalize to novel tasks. And wait isn’t that exactly what LLM training is for? It is, and it’s why we can’t use in-context learning continually. The extra pieces we need to make it work, roughly map to what LLM training already does, and we don’t know how to do that continually!
Humans are Incredibly Sample-Efficient
So any continual learning needs to be stored in the LLM parameters, which leaves us with a gap of 3 to 6 orders of magnitude in sample efficiency. How do we overcome this gap? For comparison, it took ~20 years for the cost of genome sequencing to move by ~5 orders of magnitude. And the No Free Lunch theorem tells us there’s no hope for a magic algorithm that universally bridges the gap and can learn anything we throw at it.[8] Improvement will only come from buying efficiency at the cost of generality, using inductive priors in the learning algorithm that exploit some structure in the training data.
But humans are general learners! How do we do it? Evolution has baked inductive priors into our learning algorithm that are matched to our environment.[9] We’re not actually general learners, just good at learning most of what we encounter. Remember the sick rats? They didn’t form the same distaste when the experiment was performed with “bright-noisy” water instead of sweet water. Some inductive prior triggered by “sweet yet harmful” ramped up the learning rate for that experience, but not when the water was switched out. The mechanism is selective and tuned by the machinery of evolution.
Sample efficiency and inductive priors go hand-in-hand. I wrote about how AlphaFold used specialized inductive priors to learn how to fold proteins from the relatively small protein database. Its priors were static and designed by specialists just for protein folding. Humans have analogous static priors that, for example, make us attend to (and learn from) snake and spider images faster than the modern equivalent categories like guns or electrical outlets. But static priors for avoiding dangerous animals we evolved with don’t help a modern human learn how to operate an iPad with only a few demonstrations. And yet human sample efficiency extends well past tasks that evolution could optimize for, from operating iPads to abstract math. Whatever inductive prior machinery powers this, it must be adaptive enough to extend efficiency to tasks our evolution could never prepare for.
Is Sample Efficiency All You Need?
Let’s assume we can bestow the magic recipe that makes human sample efficiency on LLMs. Could we then continue training forever, and update the parameters of a user or firm’s dedicated LLM as new observations roll in?
It turns out that the differences between humans and LLMs don’t stop at inductive priors and sample efficiency. If we continued training an LLM, even a sample-efficient one, it would eventually lose plasticity[10] and stop learning anything at all. Even worse, new observations would start to eat what the LLM learned from the old ones well before loss of plasticity kicks in. It’s called catastrophic interference,[11] and the standard mitigation is to randomize the order of the training data. If the data is random, no concept or topic is bunched up at a single point during training to be wiped out by later data. Here’s an excerpt from a real training batch (a group of sequences the LLM trains on simultaneously):
Learning like this would be as if your kindergarten curriculum were made up of quantum mechanics, the history of woodworking, and the alphabet, all taught in random order.
In a continual learning setting, we don’t get to pick the order of the observations. If we ignore randomization and just continue training with new observations as they come, we’d interfere with and degrade what the LLM already learned. If we fight this catastrophic interference by shuffling the handful of new observations in with the massive original training data, the tiny volume of new observations would be completely overwhelmed. The LLM would learn very little from them.
But let’s just assume we solve both sample efficiency and catastrophic interference. Would we then have a model that learns continually from a user or firm, picking up all the tacit knowledge today’s LLMs lack?
Yes, but it’s worth pointing out a few practical challenges with the premise. Frontier LLMs have parameters in the trillions, and moving them about in datacenters takes time and power. The latest Kimi K3 flagship release has 2.8 trillion of them. Freezing them after training means one copy can serve everyone. Your Claude prompt gets grouped together with others into a batch and the GPU re-uses the same frozen parameters when generating outputs for all of them. One startup is even etching the parameters directly into silicon.
Ignoring all of this and assuming we can figure out an efficient way to serve personalized parameters to every user, we could continue to train this idealized model on new observations and solve continual learning, just like in humans. So how do we solve catastrophic interference, and how do we make model training sample-efficient?
Learning to Walk Before You Run
How did humans solve continual learning? Does that tell us anything about how to solve it for LLMs? We’ve seen that humans owe their sample efficiency to inductive priors. But unlike AlphaFold whose specialized and static priors give it efficiency in a narrow domain, humans are generally sample-efficient. For example, a surgeon learns the tacit knowledge of their field from their first-hand experience in the role.
How would inductive priors for surgery even work? We can be sure there aren’t any encoded in the human genome. One clue is in the difference between how LLMs and humans need their training data ordered. LLMs train on randomly ordered data to prevent catastrophic interference. Humans in contrast need strict ordering to learn the prerequisites first. There’s no way to learn surgery without first learning anatomy. If you try, the anatomical parts of it will be incomprehensible. If we view anatomy as a learned inductive prior for surgery, we could imagine how humans keep their sample efficiency at the expense of needing concepts to be learned in their compositional order, from simple to complex.
There’s evidence of compositional learning of priors in animals, albeit for simpler tasks than surgery. Rats (again) that had already learned the “schemas” for flavor-place association tasks could learn new pairs in a single trial, a process that would otherwise take weeks.[12]
“systems consolidation can occur extremely quickly if an associative ‘schema’ into which new information is incorporated has previously been created… New traces, trained for only one trial, then became assimilated and rapidly hippocampal-independent.”
So why don’t we do this with LLMs? Why not learn concepts in order, and have the LLM bootstrap more complex concepts by using their prerequisites as inductive priors? Because catastrophic interference wipes out earlier concepts when training the more complex ones.
If we solved catastrophic interference, would existing transformer-based LLM architectures automatically use prerequisites as priors to efficiently learn complex concepts? The answer is maybe. We at least know that training LLMs with a curated curriculum of topics ordered compositionally does not work as well as training them with data in random order.[7] The deficiency could be simply a matter of solving catastrophic interference, but it’s unclear whether further problems await beyond that. I could not find an experiment that disentangles the two and uncovers whether LLMs use past knowledge as inductive priors for future training. My guess is that the two problems have different shapes and require different solutions, but I hope I’m wrong.
Beyond Centralized Training
Xerox’s attempt to build an AI to help their technicians diagnose machine faults failed in part because it didn’t learn alongside them. What the technicians needed was help with new problems, not AI trained on the spec and documentation they already had. What worked with 90s-era technology was not an AI at all, but a system for sharing tips to improve the technicians’ own continual learning. Xerox’s problem persists today: the tacit knowledge necessary to contribute to firms is unique, generally not written down, and must be learned from a few samples. Only humans can do this right now.
We live in the centralized AI training era. LLMs are sample-inefficient and difficult to train, so training is done once, carefully, and with massive curated datasets. But the economy is vast and to contribute to its many firms you need tacit, firm-specific knowledge that’s not written down, and needs to be learned from just a few examples. For AI’s impact on science, it needs to learn from new experiments to go beyond existing science, just like humans would. It can’t do this right now and the problems barring it are research-shaped, but humans and their efficient continual learning give us some hints about what the solutions could look like.
Brynjolfsson & Hitzig, AI’s Use of Knowledge in Society.
^Boris Groysberg, Linda-Eling Lee & Ashish Nanda, Can They Take It With Them? The Portability of Star Knowledge Workers’ Performance, Management Science 54(7), 2008, 1213–1230. Extended in Chasing Stars (Princeton University Press, 2010).
^Matthew Bidwell, Paying More to Get Less: The Effects of External Hiring versus Internal Mobility, Administrative Science Quarterly 56(3), 2011, 369–407.
^Daniel G. Bobrow & Jack Whalen, Community Knowledge Sharing in Practice: The Eureka Story, Reflections (SoL Journal) 4(2), 2002. academia.edu.
^John Garcia & Robert A. Koelling, Relation of Cue to Consequence in Avoidance Learning, Psychonomic Science 4, 1966, 123–124. DOI:10.3758/BF03342209.
^Michael C. Frank, Bridging the Data Gap Between Children and Large Language Models, Trends in Cognitive Sciences 27(11), 2023, 990–992. DOI:10.1016/j.tics.2023.08.007.
^Alex Warstadt et al., Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora, CoNLL 2023, 1–34. babylm.github.io.
^David H. Wolpert, The Lack of A Priori Distinctions Between Learning Algorithms, Neural Computation 8(7), 1996, 1341–1390. DOI:10.1162/neco.1996.8.7.1341.
^Elizabeth S. Spelke & Katherine D. Kinzler, Core Knowledge, Developmental Science 10(1), 2007, 89–96. DOI:10.1111/j.1467-7687.2007.00569.x.
^Baekrok Shin, Junsoo Oh, Hanseul Cho & Chulhee Yun, DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticity, NeurIPS 2024. arXiv:2410.23495.
^Timothée Lesort, Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes, PhD thesis, 2020. arXiv:2007.00487.
^Dorothy Tse et al., Schemas and Memory Consolidation, Science 316(5821), 2007, 76–82. DOI:10.1126/science.1135935.
^
I would very much like to convince you that this is not a helpful topic to explore.
I have thought about this question a lot over 23 years of studying the computational and psychological mechanisms of human learning. I’ve been resisting writing about my ideas on this subject for around 10 years now. I feel even more strongly now that good ideas in this area may accelerate AI progress at a time when it’s already moving too fast for safety by almost any careful estimate.
Thank you for the comment, Seth. I’m very open to a discussion and a belief update. The reason I am interested and in and optimistic about continual learning is because I think it allows smaller, more personalized models to be successful. A few reasons that I touched on briefly in the essay:
Large (and getting larger) models are efficient to deploy in part because their weights are frozen. Deploying continual learning is small-model territory. Even as compute and memory availability expand, the delta in deployment difficulty between frozen and continually learning weights will remain.
Large, centrally trained models not only suffer from mode collapse themselves, they mode-collapse the humans that use them. If we use models that all know the same things and think the same way, what happens to the pluraly of human thought and culture?
Broader economically productive deployment of smaller and more personalized models (downstream of continual learning) will disincentivize training ever larger models. Especially if limits (as I’ve argued) to ROI are no longer on raw knowledge/intelligence/agency but rather missing tacit knowledge.
Arguments/ideas about what would make smaller models catch up to the frontier should confront what happens when you use the same improved methods for the biggest models. Usually, it doesn’t narrow the gap, because both things get stronger.
And also the bigger models get even stronger, which is not a good thing at the current pace of progress. Though I think it’s worth distinguishing within-paradigm progress that will plateau in approximately the same place, provided the current paradigm plateaus, from paradigm-breaking progress that might let progress escape some expected plateau. Faster within-paradigm progress (if it was easy to confidently judge as such) might even be a good thing, since it reduces the overhang, and so reduces the pace of subsequent progress to the pace of scaling. But ambitious continual learning that lets models automatically learn deep skills quickly (as opposed to gaining an effectively unbounded context while remaining constrained by the effective fidelity of in-context learning) is likely paradigm-breaking, so it’s clearly not an ambiguously safe thing to pursue.
It’s not clear to me that large models getting better with CL means the adoption (not capability, as measured by centralized benchmarks) gap between them and small models will hold. First, small models will get more benefit than large models. But more importantly, if the extra capabilities of the larger model are not economically meaningful against the costs, then the market will select the smaller model. A historical analog is the application of Moore’s law to both mainframes and PCs. Both got better, but the market selected the lower-cost, right-sized, and personalized option.
Why is that? Bigger models are better at making use of any given amount of data, so they might benefit more, if there’s enough compute to go around.
In some ways, the modern 1 GW datacenters beg to differ.
The argument assumes that frontier models are already capable enough (“PhD level”) for most general whilte collar economic tasks that they’re being applied to, and that what’s missing is the tacit knowledge enabled by CL. Making bigger models bigger just won’t fix this blocker.
Yes, and to use the analogy, it’s because we’re in the mainframe era. Once CL is available, it would be akin to the introduction of the PC.
Mainframes never went away. They became data centers, which are even more centralized, and got bigger. The tasks that could be offloaded to small local computers were, and the ones that couldn’t, didn’t, and we kept inventing new jobs for both as both got stronger.
So far I see AI following the same path.
A mainframe is a single large very powerful machine. A datacenter is a bunch of servers (the PC model) that are conveniently colocated. I think the analogy is consistent with data centers.
GPU datacenters are different and closer to one giant machine, but that’s a recent phenomenon with centralized training of AI.
It’s worth distinguishing big datacenters (especially for pretraining, where scale-out networks need to be unusually good) from big scale-up systems/pods. Big scale-up systems (currently mostly rack-level, though TPUs were multi-rack for many years now) are motivated by very big numbers of total params in MoE models, not by centralized training of AI. Centralized training of AI (in the sense of pretraining) instead motivates big datacenters with good scale-out networks, but the individual scale-up systems within these datacenters could be small (even for models with a lot of total params). So these are completely different desiderata, calling for technologically unrelated things.
That’s a good point, regarding the CPU datacenters (though I don’t see how the analogy could transfer something useful at this level, if the conclusion isn’t already accepted and the analogy just illustrates what it looks like based on a more familiar story).
The question was what happens to bigger vs. smaller models when we do fix this blocker for both. You can’t just apply the improved method to the smaller models, and compare that to the bigger models without the improved method (whether the bigger models in fact get trained depends on the answer to this question about the capability consequences of hypothetically training the improved bigger models regardless of their presumed economic usefulness). So the relevant thing is whether the bigger models can still make better use of any given amount of data (continual learning or not), and thus whether the gap between the smaller and the bigger models persists.
They still aren’t capable enough at whatever they aren’t in fact capable enough for. And those further things can have enormous TAM, all the way to taking over the world and the reachable universe. This is only irrelevant for the question of big vs. small models if the gap in fact disappers.
If the gap remains, while the smaller models are capable enough for most general economic tasks, then the bigger models will be even more capable than that. The crux then shifts to whether this is even possible (but then it can’t affect the argument itself, that would be rationalization from the bottom line back to the argument), or whether it’s valuable to be significantly more capable than whatever most modern economic tasks require.
I tried to address this but might have been missed:
So both get a benefit, but if what was blocking economic ROI was CL, and not the extra capabilities of the larger models, then smaller models will win.
Yes, and I think we’ve established the “further thing” is CL (at least in the context of this argument), so then the enourmous TAM is tied to CL, not how big the model is. You would obviously want to own the weights, and you’d want to serve the smallest possible model that delivered the returns of the CL to your org.
That extra capability only matters if it’s economically useful. Like I said before (and analogized with the mainframe vs. PC argument), the model being able to solve math conjectures is not useful for the white collar work that has repeatedly resisted automation with AI. More of that will not change the equilibrium.
I’m not even insisting that the extra capabilities matter. I’m insisting that you didn’t argue that they aren’t there (capabilities of big over small models, given CL or whatever other improvements). Or that the capability (rather than adoption) gap gets smaller. Whether the bigger models are economically useful is downstream of whether they’re importantly more capable, the question of relative capability is a key input to the outcome of adoption, while the question of adoption doesn’t inform the question of relative capability at all.
That’s why I mentioned the reachable universe. Some amount of ASI-pilledness is necessary for a reasonable discussion about what happens when the bigger models saturate the status quo level of capabilities of the modern humanity.
What I said was
The “further things” in my intended meaning are capabilities (and the accomplishment of the more difficult tasks), not methods.
CL may give small models a relative advantage, but it will benefit large models almost as much. That acceleration will make an already very dangerous situation more dangerous. We are getting AI progress way faster than is safe.
What happens to the plurality of thought and culture is just not on my long list of things to worry about. Massive job loss and resulting economic collapse barely makes it. Dismissing the alignment problem is simply not reasonable. We don’t know how hard alignment is, but assuming it’s not a problem just isn’t reasonable.
And racing toward AGI/ASI makes that problem a lot harder. Accelarating that race just isn’t responsible.
And CL raises a bunch more alignment issues: see How might continual learning affect safety and alignment? OTOH I think we’ll need to tackle those sooner or later; but thinking that adding it sooner is an alignment advantage sounds like motivated reasoning allowing me to publish some fun and important ideas.
How would they do that? Not without destroying the current prospects for AI company growth. I estimate that the costs per token of even quadrillion total param models of 2031 enable serving them at a price of $30/$150 per 1M input/output token for a 70% gross margin (provided the long term contract pricing on compute remains in the current ballpark), or $15/$75 for a 40% gross margin. This is from a 10 GW scale pretraining run, which seems reasonable if AI companies end up with 30-50 GW of compute individually by late 2030 (counting both long term contracts usable for training and TaaS contracts that are only good for inference).
Because the marginal benefit from a smaller model learning your company’s tacit knowledge far outweights the marginal benefit from training a model with larger parameters. Like your marketing agent just doesn’t benefit from GPT6 solving yet another math conjecture. It does benefit from knowing your business and your customers and the subtleties of your particular industry.
Yes, and I don’t think AI company growth prospects are a universal given. If we do end up with CL models, companies likely won’t want the learned knowledge to be distributed with other users of the model. So the incentive will be to own the weights, which directly threatens the business of the frontier model companies and their capacity to centrally train ever-larger models.
The crux is that bigger models are not very expensive. If bigger models are importantly more capable, if wouldn’t matter if they are moderately more expensive; and if they are only marginally more capable (even for the most difficult tasks), it doesn’t matter if they get trained (they might still in fact get trained). Prospects for AI company growth don’t obviously suffer even if bigger models aren’t important, because established AI companies can remain good at making smaller models as well.
And if AI companies do grow (for the biggest companies to individually command about 10-20% of global compute, out of hundreds of gigawatts), this then determines if the bigger models get trained. The fixed costs of training won’t be very high compared to the AI company scale, and they are still not very expensive so they’ll have their uses (even if they aren’t much more capable for many easier applications), and their dangers. Distillation motivates making very big models even when they are not directly useful, to make the smaller models marginally more capable, but the biggest models still not being very expensive makes it likely that they are available directly.
In addition to what others have said, I’m not sure it would be good from a safety perspective if smaller, personalized models became more viable. A big part of what makes current governance proposals anywhere close to workable is the fact that big models need big datacenters to train, so by restricting the datacenters you can restrict the big training runs. If it were possible to train AGI on consumer-grade hardware, it would be much harder to coordinate to slow down or pause development if that becomes necessary (I’d argue it’s already necessary but that’s beside the point).
Without CL, the main axis the model companies compete on is larger, more broadly capable models. The point I made in the other threads is that if you’re a SMB and need a slack AI to manage customer service, you don’t need that AI to also know organic chemistry. That it does today is an economic abberation simmilar to the adoption of mainframes before the PC came around. The PC decimated mainframes not because it was more powerful, but because it was cheaper, right-sized, and personalized.
If it’s true that CL is the missing capability for businesses to get ROI on AI, and not marginal improvement on group theory, or nuclear engineering, or biology, then its introduction will dampend incentives to ship ever larger models with “dangerous” knowledge very few businesses actually need, even if those larger models would similarly benefit from CL.
Technical details of token costs say it’s active params that determine costs, not total params (when running within sufficiently big scale-up systems). Knowing more things (as opposed to being smarter about them) is more a total params thing (and currently there is no way to avoid knowing specific things; you can only avoid putting the effort into getting even better at them than you would by default). If there is some niche application that benefits from knowing organic chemistry, and the AI company puts in the work to get some model good at it, then there is no reason not to make all the models it makes good at it, to the extent they are able to stay coherent about sufficiently complicated questions like that (given the number of active params their cost permits them to have).
So there is no technical reason for there being an economic benefit from knowing less, if an AI company already knows how to teach AIs those things. Even cheap models with few active params can receive all the training in organic chemistry, they might just remain unable to get very good at it, but judging by the current capabilities even small cheap models will still be better than non-specialist humans. Future bigger scale-up systems/pods will enable more total params, to the point that in mid-2030s there might be “small” quadrillion param models that are very cheap (if very high levels of sparsity are technically feasible, which they might be with an absurd total number of reasonably sized routed experts).
Currently, models with fewer total params are useful because there are many legacy servers that can’t run the big models at all, not because they are inherently cheaper to run on the new bigger servers (which in 2026 are mostly rack-level scale-up systems like GB300 NVL72, but the future is multi-rack). This consideration will eventually go away, because bigger servers/pods are made out of the same stuff, the technology for connecting them into a single whole (scale-up system) just didn’t catch up to the demands of frontier LLMs yet, and therefore it’s not available for cheap models with few active params either, which is why cheap models with few active params also tend to have fewer total params.
I think you’re still assuming there’s this one megacompany that’s training a single model distributed to the whole world. A small model can be trained by a specialized company that’s a good enough bootstrap based for a basket of skills such that CL can then specialize it for a user environment. If that small model can take market share for applications that require that basket of skills away from the mega AI corp, then you have the PC/mainframe dynamic again where the mainframe play gets completely disrupted.
My technical point is that “a small model” equivocates between active and total params, and a small-meaning-cheap model might have a quadrillion of total params in mid-2030s. Even with a few trillion total params, it can know all the things. Small-meaning-cheap 50B-100B active param models with trillions of total params are already happening; though such models are not yet the cheapest models, because GB300 NVL72 is not yet the low-end kind of hardware. So saying that a customer service model knowing organic chemistry is an abberation that will go away needs the premise that the company training the customer service model has difficulty obtaining the training data/process for organic chemistry, or vice versa, because there is no downside at the unit economics stage of the model’s lifecycle.
Because of the non-negative (and probably positive) transfer, meaning knowing more in one domain doesn’t hurt model performance in the other domains, and no effect on unit costs because of the active/total params distinction, it’s useful to make models that know all of the things simultaneously, for any target price tier. So there is a natural pressure for AI companies that specialize in different domains (to the extent such a thing happens at all) to merge, or to never seriously develop separately past a startup-acquisition phase (or beyond data-sourcing activities, selling the data to the companies actually training the models). This is a consequence of the unusual technological properties of MoE models, which don’t hold for many other kinds of products. If anything, CL makes it easier for a poorly focused AI megacorp to enter niche domains, which it doesn’t then need to know in subtle detail, letting its AIs get to know these domains on their own instead.
Hmm, I kind of suspect that there’s something deep about the fact that making models larger makes them smarter. Something like, I have a prior that all knowledge is connected, such that in principle getting better at nuclear engineering does make you better at, like, biology, even if only a little. Maybe you’ll say “okay, but with CL companies could make models that are good enough at what they need while still being non-general”. Sure, maybe, but the worry is that if you can make a model like that on consumer grade hardware, then eventually you could also make a general “AGI” on consumer-level hardware if you wanted to, and that would be dangerous for all the usual reasons. And if that’s possible, it’s much harder to stop than the current large training runs for the reasons I mentioned in my previous comment.
I’d very much appreciate a discussion on the argument.
Some ways I could see pushing back in it:
Humans learn better out of order: there are studies showing this, but as far as I could find the material being shuffled were not compositionally depenedent, meaning they could be learned in any order.
Better architecture, not priors: in a previous article on AlphaFold, I argued that its hard-coded inductive priors lent it the sample-efficiency it needed to learn something generalizable for the relatively sparse protein database. There, the priors were baked in a specialized architecture, but humans are sample-efficient more broadly. So perhaps there is an architecture that is just more sample efficient without needing ordered training or hierarchical priors.
What about replay: our brain does replay experiences, which nominally mitigates catastrophic interference. Although the replay seems to prioritize surprise/novelty rather than already-learned experiences.