Regarding continual learning and memory, I mentioned in the dialogue that I’m not just talking about performance of trained LLMs, but rather addressing the whole Architecture:
When an LLM is trained more, it gains more partial concepts.
However, it gains more partial concepts with poor sample efficiency; it mostly only gains what’s in the data.
In particular, even if the LLM were being continually trained (in a way that’s similar to how LLMs are already trained, with similar architecture), it still wouldn’t do the thing humans do with quickly picking up new analogies, quickly creating new concepts, and generally reforging concepts.
For example, sometimes people believe that, for some X, we just need X to make AGI from current ML systems. Sometimes they believe this because they are imputing the ghost in the machine. E.g.: “LLMs don’t get feedback from the environment, where they get to try an experiment and then see the results from the external world. When they do, they’ll be able to learn unboundedly and be fully generally intelligent.”. I think what this person is doing is imagining themselves without feedback loops with external reality; then imagining themselves with feedback loops; noticing the difference in their own thinking in those two hypotheticals; and then imputing the difference to the LLM+feedback system, imagining that the step LLM⟶ LLM+feedback is like the step human⟶ human+feedback. In this case imputing the ghost is a mistake in both ways: they don’t realize that they’re making that imputation, and the LLM+feedback system actually doesn’t have the imputed capabilities. They’re falsely imputing [all those aspects of their mind that would be turned on by going from no-feedback to yes-feedback] to the LLM+feedback. That’s a mistake because really the capabilities that come online in the human⟶ human+feedback step require a bunch of machinery that the human does have, in the background, but that the LLM doesn’t have (and the [LLM+feedback + training apparatus] system doesn’t have the machinery that [human + humanity + human evolution] has).
Note that I was talking about both long-term memory and continual learning, not just continual learning, so I’m happy to concede that my proposed architecture is not like how LLMs are trained today, and thus could reasonably be called a non-LLM architecture.
Though I will say that the BabyLM challenge and to a lesser extent the connect the dots paper is evidence that part of the reason current LLMs are so data inefficient is not because of fundamental limitations, but rather because AI companies didn’t really need to have LLMs be data efficient in order for LLMs to work so far, but by 2028-2030, this won’t work nearly as effectively assuming LLMs haven’t automated away all AI research.
You’ve mentioned the need for a missing update, and I think part of that missing update is that we didn’t really realize how large the entire internet was, and this gave the fuel for the very impressive LLM scaling, but this is finite, and could very plausibly not be enough for LLMs out of the current companies.
However, I’m inclined towards thinking the issue may not be as fundamental as you think it is, for the reason @abramdemski said below:
My earlier perspective, which asserted “LLMs are fundamentally less data-efficient than humans, because the representational capabilities of Transformers aren’t adequate for human concepts, so LLMs have to memorize many cases where humans can use one generalization” would have predicted that it is not possible to achieve GPT2 levels of linguistic competence on so little data.
Given the budgets involved, I think it is not at all surprising that only a GPT2 level of competence was reached. It therefore becomes plausible that a scaled-up effort of the same sort could reach GPT4 levels or higher with human-scale data.
The point being: it seems to me like LLMs can have similar data-efficiency to humans if effort is put in that direction. The reason we are seeing such a drastic difference now is due more to where the low-hanging fruit lies, rather than fundamental limitations of LLMs.
Remember, this is a small scale experiment, and you often have to go big in order to make use of your new findings, even if there are enough efficiency tricks such that at the end, you can make an AI that is both very capable and more efficient than modern human learning (I’m not assuming that there exists a method such that LLMs can be made more data efficient than a human, but am claiming that if they exist, there still would need to be scaling to find those efficiency tricks).
So it being only as good as GPT2 is unsurprising. Keep in mind that GPT-3 was trained by OpenAI who absolutely believed in the ability to scale up compute, and had more resources than academic groups at the time.
For example, sometimes people believe that, for some X, we just need X to make AGI from current ML systems. Sometimes they believe this because they are imputing the ghost in the machine. E.g.: “LLMs don’t get feedback from the environment, where they get to try an experiment and then see the results from the external world. When they do, they’ll be able to learn unboundedly and be fully generally intelligent.”. I think what this person is doing is imagining themselves without feedback loops with external reality; then imagining themselves with feedback loops; noticing the difference in their own thinking in those two hypotheticals; and then imputing the difference to the LLM+feedback system, imagining that the step LLM⟶ LLM+feedback is like the step human⟶ human+feedback. In this case imputing the ghost is a mistake in both ways: they don’t realize that they’re making that imputation, and the LLM+feedback system actually doesn’t have the imputed capabilities. They’re falsely imputing [all those aspects of their mind that would be turned on by going from no-feedback to yes-feedback] to the LLM+feedback. That’s a mistake because really the capabilities that come online in the human⟶ human+feedback step require a bunch of machinery that the human does have, in the background, but that the LLM doesn’t have (and the [LLM+feedback + training apparatus] system doesn’t have the machinery that [human + humanity + human evolution] has).
My main response is that once we condition on LLMs not having weight level continual learning as well as them not having a long-term memory, there’s little mystery left to explain for LLM capabilities, so there’s no other machinery that I’ve missed that is very important.
For the continual learning point, a great example of this is that humans don’t hit walls of capability nearly as often as LLMs do, and in particular human success curves on task often flatline or increase, rather than hit hard limits, and in particular when needed have very, very high conceptual resolution, such that we can work on long, open-ended problems without being entirely unproductive of insights.
And this is because human neurons constantly update, and there’s no deployment phase where all your neurons stop updating.
Human neuroplasticity declines, but never is completely gone as you age.
I explain more about why I think continual learning is important below, and @gwern really explained this far better than I can, so read Gwern’s comment too:
For the long-term memory point, the reason why it’s important for human learning is that it simultaneously prevents us from being stuck on unproductive loops like how Claude can go to the Spiritual Bliss attractor or how Claude has a very bad habit of being stuck in loops when trying to win the game of Pokemon, and also allows you to build on previous wins/take in large amounts of context without being lost, which is a key part of doing jobs.
Dwarkesh Patel explains better than I can why a lack of long-term memory/continual learning is such a big deal for LLMs, and reduces their ability to be creative, because they cannot build upon hard-earned optimizations into something bigger, and I tend to model humans getting insights not as you thinking hard for a day and fully forming the insight like Athena out of Zeus, but rather humans getting a first small win, and because they can rely on their long-term memory, they don’t have to worry about losing that first small win/insight, and they continuously look both at reality and theory to iteratively refine their insights until they finally have a big insight that comes out after a lot of build-up, but LLMs can’t ever build up to big insights, because they keep constantly forgetting the small stuff that they have gotten:
LLMs actually do get kinda smart and useful in the middle of a session. For example, sometimes I’ll co-write an essay with an LLM. I’ll give it an outline, and I’ll ask it to draft the essay passage by passage. All its suggestions up till 4 paragraphs in will be bad. So I’ll just rewrite the whole paragraph from scratch and tell it, “Hey, your shit sucked. This is what I wrote instead.” At that point, it can actually start giving good suggestions for the next paragraph. But this whole subtle understanding of my preferences and style is lost by the end of the session.
Maybe the easy solution to this looks like a long rolling context window, like Claude Code has, which compacts the session memory into a summary every 30 minutes. I just think that titrating all this rich tacit experience into a text summary will be brittle in domains outside of software engineering (which is very text-based). Again, think about the example of trying to teach someone how to play the saxophone using a long text summary of your learnings. Even Claude Code will often reverse a hard-earned optimization that we engineered together before I hit /compact—because the explanation for why it was made didn’t make it into the summary.
Edit: And the cases where there are fully formed ideas from what seems like to be nothing is because of your default mode network in the brain, and more generally you always are computing and thinking in the background, and it’s basically continual learning on your own thoughts, and once we realize this, it’s much less surprising that we can somewhat reliably create insights. LLMs lack any equivalent of a default mode network/continual learning on their own thoughts, which pretty neatly explains why people report that they have insights/creativity out of nowhere, but LLMs so far haven’t done this yet:
Another edit: A key part of my worldview is that by the 2030s, we have enough compute such that we can constantly experiment with human-brain sized architectures, and in particular given that I think capabilities are learned within life-time for various reasons @Steven Byrnes and @Quintin Pope already said, and this means that the remaining missing paradigms are likely to be discovered more quickly, and critically this doesn’t depend on LLMs becoming AGI:
An important study here that’s quoted for future reference:
October 2024: I found a paper Blumberg & Adolph 2023, which discusses the extent to which the cortex is involved in newborn behavior. Their answer is “very little” (which supports my hypothesis). I added it as a reference in Section 2.5.2.
A key crux that I hold, relative to you is that I think LLMs are in fact a little bit creative/can sometimes form insights (though with caveats), but that this is not the relevant question to be asking, and I think most LLM incapacities are not literally that they can never do this fundamentally, but rather that at realistic amounts of compute and data, they cannot reliably form insights/be creative on their own, or even do as well as the best human scientists, similar to @Thane Ruthenis’s comment below:
So the lack of long-term memory and continual learning is closer to the only bottleneck for LLMs (and I’m willing to concede that any AI that solves these bottlenecks is not a pure LLM).
Also, this part is something that I agree with, but I expect normal iteration/engineering to solve these sorts of problems reliably, so I don’t consider it a fundamental reason not to expect AGI in say the 2030s:
Most instances of a category are not the most powerful, most general instances of that category. So just because we have, or will soon have, some useful instances of a category, doesn’t strongly imply that we can or will soon be able to harness most of the power of stuff in that category.
Remember, this is a small scale experiment, and you often have to go big in order to make use of your new findings,
Ok… but are you updating on hypothetical / fictional evidence? BTW to clarify, the whole sample efficiency thing is kind of a sideline to me. If someone got GPT4 level performance by training on human data that is like 10x the size of the books that a well-read human would read in 50 years, that would be really really weird and confusing to me, and would probably shorten my timelines somewhat; in contrast, what would really shorten my timelines would be observations of LP2X creating novel interesting concepts (or at least originary interesting concepts, as in Hänni’s “Cantor’s Diagonal from scratch” thing).
My main response is that once we condition on LLMs not having weight level continual learning as well as them not having a long-term memory, there’s little mystery left to explain for LLM capabilities, so there’s no other machinery that I’ve missed that is very important.
Yes, this is a good example of mysteriously assuming for no reason that “if we just get X, then I don’t see what’s stopping my learning program from being an AGI, so therefore it is an AGI”, which makes absolutely zero sense and you should stop.
And this is because human neurons constantly update, and there’s no deployment phase where all your neurons stop updating.
No it’s not. I mean it is a little bit. But it’s also “because” “neurons implement Bayesian learning”, and it’s also “because” “neurons implement a Turing machine”. Going from this sort of “because” to “because my thing is also a Turing machine and therefore it’s smart, just like neurons, which are also Turing machines” makes zero sense.
relative to you is that I think LLMs are in fact a little bit creative/can sometimes form insights (though with caveats)
What considerations (observations, arguments, etc.) most strongly contributed to convincing you of the strongest form of this proposition that you believe?
I do not think that Noosphere’s comment did not contain an argument. The rest of the comment after the passage you cited tries to lay out a model for why continual learning and long-term memory might be the only remaining bottlenecks. Perhaps you think that this argument is very bad, but it is an argument, and I did not think that your reply to it was helpful for the discussion.
Regarding continual learning and memory, I mentioned in the dialogue that I’m not just talking about performance of trained LLMs, but rather addressing the whole Architecture:
Your remarks sound to me like “We just need X”, which I addressed here: https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#_We_just_need_X__intuitions
See also https://tsvibt.blogspot.com/2023/09/a-hermeneutic-net-for-agency.html#silently-imputing-the-ghost-in-the-machine , which I’ll quote from:
Note that I was talking about both long-term memory and continual learning, not just continual learning, so I’m happy to concede that my proposed architecture is not like how LLMs are trained today, and thus could reasonably be called a non-LLM architecture.
Though I will say that the BabyLM challenge and to a lesser extent the connect the dots paper is evidence that part of the reason current LLMs are so data inefficient is not because of fundamental limitations, but rather because AI companies didn’t really need to have LLMs be data efficient in order for LLMs to work so far, but by 2028-2030, this won’t work nearly as effectively assuming LLMs haven’t automated away all AI research.
You’ve mentioned the need for a missing update, and I think part of that missing update is that we didn’t really realize how large the entire internet was, and this gave the fuel for the very impressive LLM scaling, but this is finite, and could very plausibly not be enough for LLMs out of the current companies.
However, I’m inclined towards thinking the issue may not be as fundamental as you think it is, for the reason @abramdemski said below:
Remember, this is a small scale experiment, and you often have to go big in order to make use of your new findings, even if there are enough efficiency tricks such that at the end, you can make an AI that is both very capable and more efficient than modern human learning (I’m not assuming that there exists a method such that LLMs can be made more data efficient than a human, but am claiming that if they exist, there still would need to be scaling to find those efficiency tricks).
So it being only as good as GPT2 is unsurprising. Keep in mind that GPT-3 was trained by OpenAI who absolutely believed in the ability to scale up compute, and had more resources than academic groups at the time.
To respond to this:
My main response is that once we condition on LLMs not having weight level continual learning as well as them not having a long-term memory, there’s little mystery left to explain for LLM capabilities, so there’s no other machinery that I’ve missed that is very important.
For the continual learning point, a great example of this is that humans don’t hit walls of capability nearly as often as LLMs do, and in particular human success curves on task often flatline or increase, rather than hit hard limits, and in particular when needed have very, very high conceptual resolution, such that we can work on long, open-ended problems without being entirely unproductive of insights.
And this is because human neurons constantly update, and there’s no deployment phase where all your neurons stop updating.
Human neuroplasticity declines, but never is completely gone as you age.
I explain more about why I think continual learning is important below, and @gwern really explained this far better than I can, so read Gwern’s comment too:
https://www.lesswrong.com/posts/5tqFT3bcTekvico4d/do-confident-short-timelines-make-sense#mibF9KKtuJxtnDBne
For the long-term memory point, the reason why it’s important for human learning is that it simultaneously prevents us from being stuck on unproductive loops like how Claude can go to the Spiritual Bliss attractor or how Claude has a very bad habit of being stuck in loops when trying to win the game of Pokemon, and also allows you to build on previous wins/take in large amounts of context without being lost, which is a key part of doing jobs.
Dwarkesh Patel explains better than I can why a lack of long-term memory/continual learning is such a big deal for LLMs, and reduces their ability to be creative, because they cannot build upon hard-earned optimizations into something bigger, and I tend to model humans getting insights not as you thinking hard for a day and fully forming the insight like Athena out of Zeus, but rather humans getting a first small win, and because they can rely on their long-term memory, they don’t have to worry about losing that first small win/insight, and they continuously look both at reality and theory to iteratively refine their insights until they finally have a big insight that comes out after a lot of build-up, but LLMs can’t ever build up to big insights, because they keep constantly forgetting the small stuff that they have gotten:
https://www.dwarkesh.com/p/timelines-june-2025
Edit: And the cases where there are fully formed ideas from what seems like to be nothing is because of your default mode network in the brain, and more generally you always are computing and thinking in the background, and it’s basically continual learning on your own thoughts, and once we realize this, it’s much less surprising that we can somewhat reliably create insights. LLMs lack any equivalent of a default mode network/continual learning on their own thoughts, which pretty neatly explains why people report that they have insights/creativity out of nowhere, but LLMs so far haven’t done this yet:
https://gwern.net/ai-daydreaming#continual-thinking
Another edit: A key part of my worldview is that by the 2030s, we have enough compute such that we can constantly experiment with human-brain sized architectures, and in particular given that I think capabilities are learned within life-time for various reasons @Steven Byrnes and @Quintin Pope already said, and this means that the remaining missing paradigms are likely to be discovered more quickly, and critically this doesn’t depend on LLMs becoming AGI:
https://www.lesswrong.com/posts/yew6zFWAKG4AGs3Wk/?commentId=Bu8qnHcdsv4szFib7
An important study here that’s quoted for future reference:
A key crux that I hold, relative to you is that I think LLMs are in fact a little bit creative/can sometimes form insights (though with caveats), but that this is not the relevant question to be asking, and I think most LLM incapacities are not literally that they can never do this fundamentally, but rather that at realistic amounts of compute and data, they cannot reliably form insights/be creative on their own, or even do as well as the best human scientists, similar to @Thane Ruthenis’s comment below:
https://www.lesswrong.com/posts/GADJFwHzNZKg2Ndti/have-llms-generated-novel-insights#YFfperxLEnWoomzrH
So the lack of long-term memory and continual learning is closer to the only bottleneck for LLMs (and I’m willing to concede that any AI that solves these bottlenecks is not a pure LLM).
Also, this part is something that I agree with, but I expect normal iteration/engineering to solve these sorts of problems reliably, so I don’t consider it a fundamental reason not to expect AGI in say the 2030s:
Ok… but are you updating on hypothetical / fictional evidence? BTW to clarify, the whole sample efficiency thing is kind of a sideline to me. If someone got GPT4 level performance by training on human data that is like 10x the size of the books that a well-read human would read in 50 years, that would be really really weird and confusing to me, and would probably shorten my timelines somewhat; in contrast, what would really shorten my timelines would be observations of LP2X creating novel interesting concepts (or at least originary interesting concepts, as in Hänni’s “Cantor’s Diagonal from scratch” thing).
Yes, this is a good example of mysteriously assuming for no reason that “if we just get X, then I don’t see what’s stopping my learning program from being an AGI, so therefore it is an AGI”, which makes absolutely zero sense and you should stop.
No it’s not. I mean it is a little bit. But it’s also “because” “neurons implement Bayesian learning”, and it’s also “because” “neurons implement a Turing machine”. Going from this sort of “because” to “because my thing is also a Turing machine and therefore it’s smart, just like neurons, which are also Turing machines” makes zero sense.
What considerations (observations, arguments, etc.) most strongly contributed to convincing you of the strongest form of this proposition that you believe?
@Lucius Bushnaq It’s not too combative, you’re wrong. My previous comment laid out what’s wrong with the reasoning. Then Noosphere89 wrote a big long comment that makes all the same lines of reasoning, still without giving any arguments. This is really bad epistemics, and people going around vibing hard about this have been poisoning (or rather, hijacking https://www.lesswrong.com/posts/dAz45ggdbeudKAXiF/a-regime-change-power-vacuum-conjecture-about-group-belief) the discourse for 5 years.
I do not think that Noosphere’s comment did not contain an argument. The rest of the comment after the passage you cited tries to lay out a model for why continual learning and long-term memory might be the only remaining bottlenecks. Perhaps you think that this argument is very bad, but it is an argument, and I did not think that your reply to it was helpful for the discussion.