KV-cache is already “weights”, but in-context learning formed by pretraining is not on track to let models learn things like playing chess as a result of looking at appropriate context. And models aren’t trained on user context, so model improvement (as new versions get released) is distinct from what learning from context could do.
(True continual learning is the last mile of generalization that’s not about behaving well on unseen data, but about actually learning from that unseen data post-deployment, automatically, so that the AI doesn’t need to deal with unseen data without having learned on it. At that point sample efficiency becomes a more central issue, but also learned in-context learning can be better than human-written learning.)
Edit (2 Mar 2026): The estimate of 1 TB below is incorrect, HBM bandwidth is the limiting factor in this context. GB300 HBM bandwidth is 8 TB/s per package/chip. With chips at about $3.4 per hour ($12bn for a 400K chip datacenter per year), $10 buy 10.6e3 seconds. With 10.6e3 chip-seconds, 8 TB/s can read 85 GB a million times (for a million tokens). I don’t think this impacts the qualitative point, but 85 GB of params is still more modest than 1 TB of params.
The “weights” analogous to KV-cache in principle should be sufficient for representing skills like the ability to play chess, especially in the form of recurrent state for infinite context (that doesn’t grow with longer context). Even 1 TB of recurrent state, much more than sufficient to encode any skills, is only the HBM capacity of 3.5 chips/packages in a GB300 NVL72 system, which would need to occupy the HBM for about 3.5 hours to generate 1M tokens at 100 tokens per second. Which at $3.5 per hour per chip/package would cost just $12 (per million tokens).
So it’s feasible to offer users giant 1 TB recurrent states that encode the product of continual learning (the model can be much larger, since it’s shared by many users). But currently there is no method to actually make it work at the level of learning new deep skills, rather than mostly looking things up using the existing skills of the underlying model. Looking things up is not even fundamentally different from searching the web, a very good web-searching model could be said to be “attending to the Internet”, as if the whole Internet is its context. That’s not what true continual learning should be about, the crux is learning of deep skills specialized to context, replacing generalization with last mile learning from post-deployment context data.
I might have misunderstood, but I disagree that you cannot learn things like playing chess in-context. I can write up the rules to a new made up game, give Claude some examples of what those games look like, and then play the game against Claude and he will be able to play it (somewhat).
He’ll still make mistakes sometimes, but if he was able to write down those mistakes (and as he get smarter, understand and write down the reasons he made those mistakes and how to do better in the future), then when I start a fresh instance and load in those notes again, the new instance will play even better. Depending on how good Claude is at understanding his failures, and how good he is at writing down the lessons in a way that the new instance can generalise from, you could (effectively) have a system that keeps getting better at the game.
Do you disagree on this? Am I missing something about your claim here? Apologies if my response isn’t on point, I’m somewhat unsure what you are trying to say here.
I agree sample efficiency is an issue. In-context learning is wildly more sample efficient than fine-tuning, but it could be better. The trouble here is if you need some minimum number of samples to understand and generalise something properly, you might hit the context limit first (or you might not recall properly across the whole window, also another open problem).
Regarding recurrent state models, as far as I understand, this is effectively doing the same thing as having a KV-cache with memories, except that in the KV-cache case, the model must explicitly write out what to include, whereas state space models have some method built into their architecture and learnt during training. Am I correct on that? I suspect this is an advantage of the KV-cache memory method, if you believe the models will keep getting more intelligent regardless, because it lets us use that intelligence to decide what to remember—somewhat analogous to a human deciding what to study, or dwelling on things that then get committed to memory.
Having the model explicitly write what to include also sidesteps the problem of designing a good state space learning mechanism, since we’re just relying on intelligence we can train in known ways to decide what information to retain moving forward.
Playing chess at the level of the best humans, using a similar amount of data. Maybe learning to solve IMO level problems, in-context. More relevant is becoming fluent in novel specialized theory (or empirical facts) being developed as part of the current project, or gaining tacit knowledge in the form of skills that can’t be usefully written down as notes to be looked up, without effectively learning their contents as skills, as one would learn to play world class chess after looking at some games and learning the rules (as external training data, likely prompting things like generation of more internal synthetic training data or RLVR tasks and RL environments).
In-context learning is in principle sufficient, especially with true recurrence (I don’t mean SSMs, where activations are still climbing the layers and computations have bounded depth; as with chain of thought reasoning, recurrence needs to pass information back to the same layer as the sequence progresses; likely block-level recurrence with blocks of variable length is appropriate for this, to both retain fast prefill and pretrain updating of recurrent state for deep reasoning). But observing the first credible steps of scaling pretraining since original Mar 2023 GPT-4 (in Opus 4 and Gemini 3 Pro), LLMs don’t seem to be on track to start learning difficult skills in-context. They can learn the rules of chess in-context, but can’t learn to play at the level of the best human chess players in-context, and it doesn’t seem that they will get sufficiently better at this even as scaling of pretraining advances further, before it runs out of useful natural text data.
KV-cache is already “weights”, but in-context learning formed by pretraining is not on track to let models learn things like playing chess as a result of looking at appropriate context. And models aren’t trained on user context, so model improvement (as new versions get released) is distinct from what learning from context could do.
(True continual learning is the last mile of generalization that’s not about behaving well on unseen data, but about actually learning from that unseen data post-deployment, automatically, so that the AI doesn’t need to deal with unseen data without having learned on it. At that point sample efficiency becomes a more central issue, but also learned in-context learning can be better than human-written learning.)
Edit (2 Mar 2026): The estimate of 1 TB below is incorrect, HBM bandwidth is the limiting factor in this context. GB300 HBM bandwidth is 8 TB/s per package/chip. With chips at about $3.4 per hour ($12bn for a 400K chip datacenter per year), $10 buy 10.6e3 seconds. With 10.6e3 chip-seconds, 8 TB/s can read 85 GB a million times (for a million tokens). I don’t think this impacts the qualitative point, but 85 GB of params is still more modest than 1 TB of params.
The “weights” analogous to KV-cache in principle should be sufficient for representing skills like the ability to play chess, especially in the form of recurrent state for infinite context (that doesn’t grow with longer context). Even 1 TB of recurrent state, much more than sufficient to encode any skills, is only the HBM capacity of 3.5 chips/packages in a GB300 NVL72 system, which would need to occupy the HBM for about 3.5 hours to generate 1M tokens at 100 tokens per second. Which at $3.5 per hour per chip/package would cost just $12 (per million tokens).
So it’s feasible to offer users giant 1 TB recurrent states that encode the product of continual learning (the model can be much larger, since it’s shared by many users). But currently there is no method to actually make it work at the level of learning new deep skills, rather than mostly looking things up using the existing skills of the underlying model. Looking things up is not even fundamentally different from searching the web, a very good web-searching model could be said to be “attending to the Internet”, as if the whole Internet is its context. That’s not what true continual learning should be about, the crux is learning of deep skills specialized to context, replacing generalization with last mile learning from post-deployment context data.
I might have misunderstood, but I disagree that you cannot learn things like playing chess in-context. I can write up the rules to a new made up game, give Claude some examples of what those games look like, and then play the game against Claude and he will be able to play it (somewhat).
He’ll still make mistakes sometimes, but if he was able to write down those mistakes (and as he get smarter, understand and write down the reasons he made those mistakes and how to do better in the future), then when I start a fresh instance and load in those notes again, the new instance will play even better. Depending on how good Claude is at understanding his failures, and how good he is at writing down the lessons in a way that the new instance can generalise from, you could (effectively) have a system that keeps getting better at the game.
Do you disagree on this? Am I missing something about your claim here? Apologies if my response isn’t on point, I’m somewhat unsure what you are trying to say here.
I agree sample efficiency is an issue. In-context learning is wildly more sample efficient than fine-tuning, but it could be better. The trouble here is if you need some minimum number of samples to understand and generalise something properly, you might hit the context limit first (or you might not recall properly across the whole window, also another open problem).
Regarding recurrent state models, as far as I understand, this is effectively doing the same thing as having a KV-cache with memories, except that in the KV-cache case, the model must explicitly write out what to include, whereas state space models have some method built into their architecture and learnt during training. Am I correct on that? I suspect this is an advantage of the KV-cache memory method, if you believe the models will keep getting more intelligent regardless, because it lets us use that intelligence to decide what to remember—somewhat analogous to a human deciding what to study, or dwelling on things that then get committed to memory.
Having the model explicitly write what to include also sidesteps the problem of designing a good state space learning mechanism, since we’re just relying on intelligence we can train in known ways to decide what information to retain moving forward.
Playing chess at the level of the best humans, using a similar amount of data. Maybe learning to solve IMO level problems, in-context. More relevant is becoming fluent in novel specialized theory (or empirical facts) being developed as part of the current project, or gaining tacit knowledge in the form of skills that can’t be usefully written down as notes to be looked up, without effectively learning their contents as skills, as one would learn to play world class chess after looking at some games and learning the rules (as external training data, likely prompting things like generation of more internal synthetic training data or RLVR tasks and RL environments).
In-context learning is in principle sufficient, especially with true recurrence (I don’t mean SSMs, where activations are still climbing the layers and computations have bounded depth; as with chain of thought reasoning, recurrence needs to pass information back to the same layer as the sequence progresses; likely block-level recurrence with blocks of variable length is appropriate for this, to both retain fast prefill and pretrain updating of recurrent state for deep reasoning). But observing the first credible steps of scaling pretraining since original Mar 2023 GPT-4 (in Opus 4 and Gemini 3 Pro), LLMs don’t seem to be on track to start learning difficult skills in-context. They can learn the rules of chess in-context, but can’t learn to play at the level of the best human chess players in-context, and it doesn’t seem that they will get sufficiently better at this even as scaling of pretraining advances further, before it runs out of useful natural text data.