I have a couple things to add here to the conversation that I think will help:
I think a lot of current LLM failures that @TsviBT is pointing to are downstream of 2 big issues currently, which is that current LLM weights are frozen, meaning that once they stop training, they have no way to learn anything new other than ICL, and while there’s some generalization ability, it’s also not enough for real use cases like automating AI research, and in particular LLMs have no long-term memory, meaning they have to relearn things that a human would only have to learn once, and the benchmark of Claude Plays Pokemon is a really good example of what happens when you don’t have long-term memory, and that’s due to LLMs being based on transformers rather than RNNs, where hidden states allow a form of long-term memory to occur.
The non-straw version of LLMs not being able to automate jobs from humans is that context windows don’t have to scale the way they have since 2019, because DRAM isn’t scaling exponentially, and in-context learning in practice turning out to be too weak to really replace weight-level continual learning, and their lack of memory and agency meaning that they constantly get stuck in loops, meaning they are useless for the vast majority of jobs.
I agree with this statement specifically here, and a reason for this is that I believe most human capabilities don’t come from evolution, but from within life-time learning, and the areas where evolution matters are unfortunately disproportionately AI alignment-relevent areas of the human brain, which is “(1) just one or two major shifts in training technique to be enough to get us from where we are to AI research assistants capable of major innovation”.
An independent reason to think AGI will come in the 2030s most likely, assuming LLMs do plateau, is because we are getting into the regime where we can more inexpensively experiment with human-brain level compute, and we are scaling through a lot of OOMs for compute usage, meaning that we can inexpensively experiment for insights a lot more than we currently do:
And to answer @Cole Wyeth’s question of why one human brain lifetime and not evolutionary timelines, the short answer is most of the capabilities are rederived for humans every lifetime, and evolution matters way less than we think it does.
I agree with Cole Wyeth that current LLMs are pretty bad at agency, and if we assume they don’t scale to better agency over time, I’d put much lower probability on LLMs being able to automate away the remaining bottlenecks to ASI, and it’s a reasonable hypothesis to hold (I’d put about 52% probability on this currently)
And in particular, I think the fact that LLM capability degrades way faster than humans as @abramdemski saw is tied to a lack of continual learning, and ICL not being enough currently to actually subsititute for weight-level continual learning.
And yet, I think there are good reasons to believe independently of LLMs that AGI/ASI is coming pretty soon, and I do think there’s reason to believe that timelines are probably short, even if LLMs do plateau.
To be clear, I think this is worse than a future where LLMs do just straight up scale to ASI.
Also, the entire crux is basically “does their in-context learning/generalization and creativity shown in current LLMs actually act as a proper subsitute for continual learning in weights and memory?”
The other crux is whether context windows will actually continue to scale in the way they have since 2019.
And in particular, I think the fact that LLM capability degrades way faster than humans as @abramdemski saw is tied to a lack of continual learning, and ICL not being enough currently to actually subsititute for weight-level continual learning.
I don’t agree with this connection. Why would you think that continual learning would help with this specific sort of thing? It seems relevantly similar to just throwing more training data at the problem, which has shown only modest progress so far.
The key reason is to bend the shape of the curve, and my key crux is I don’t expect throwing more training data to change the shape of the curve where past a certain point, LLMs sigmoid/fall off hard, and my expectation is more training data would make LLMs improve, but they’d still have a point where once LLMs are asked to do any task harder than that point, LLMs start becoming incapable more rapidly in humans.
To quote Gwern:
But of course, the interesting thing here is that the human baselines do not seem to hit this sigmoid wall. It’s not the case that if a human can’t do a task in 4 hours there’s basically zero chance of them doing it in 48 hours and definitely zero chance of them doing it in 96 hours etc. Instead, human success rates seem to gradually flatline or increase over time, especially if we look at individual steps: the more time that passes, the higher the success rates become, and often the human will wind up solving the task eventually, no matter how unprepossessing the early steps seemed. In fact, we will often observe that a step that a human failed on earlier in the episode, implying some low % rate, will be repeated many times and quickly approach 100% success rates! And this is true despite earlier successes often being millions of vision+text+audio+sensorimotor tokens in the past (and interrupted by other episodes or tasks themselves equivalent to millions of tokens), raising questions about whether self-attention over a context window can possibly explain it.
It seems to me like the improvement in learning needed for what Gwern describes has little to do with “continual” and is more like “better learning” (better generalization, generalization from less examples).
Regarding continual learning and memory, I mentioned in the dialogue that I’m not just talking about performance of trained LLMs, but rather addressing the whole Architecture:
When an LLM is trained more, it gains more partial concepts.
However, it gains more partial concepts with poor sample efficiency; it mostly only gains what’s in the data.
In particular, even if the LLM were being continually trained (in a way that’s similar to how LLMs are already trained, with similar architecture), it still wouldn’t do the thing humans do with quickly picking up new analogies, quickly creating new concepts, and generally reforging concepts.
For example, sometimes people believe that, for some X, we just need X to make AGI from current ML systems. Sometimes they believe this because they are imputing the ghost in the machine. E.g.: “LLMs don’t get feedback from the environment, where they get to try an experiment and then see the results from the external world. When they do, they’ll be able to learn unboundedly and be fully generally intelligent.”. I think what this person is doing is imagining themselves without feedback loops with external reality; then imagining themselves with feedback loops; noticing the difference in their own thinking in those two hypotheticals; and then imputing the difference to the LLM+feedback system, imagining that the step LLM⟶ LLM+feedback is like the step human⟶ human+feedback. In this case imputing the ghost is a mistake in both ways: they don’t realize that they’re making that imputation, and the LLM+feedback system actually doesn’t have the imputed capabilities. They’re falsely imputing [all those aspects of their mind that would be turned on by going from no-feedback to yes-feedback] to the LLM+feedback. That’s a mistake because really the capabilities that come online in the human⟶ human+feedback step require a bunch of machinery that the human does have, in the background, but that the LLM doesn’t have (and the [LLM+feedback + training apparatus] system doesn’t have the machinery that [human + humanity + human evolution] has).
Note that I was talking about both long-term memory and continual learning, not just continual learning, so I’m happy to concede that my proposed architecture is not like how LLMs are trained today, and thus could reasonably be called a non-LLM architecture.
Though I will say that the BabyLM challenge and to a lesser extent the connect the dots paper is evidence that part of the reason current LLMs are so data inefficient is not because of fundamental limitations, but rather because AI companies didn’t really need to have LLMs be data efficient in order for LLMs to work so far, but by 2028-2030, this won’t work nearly as effectively assuming LLMs haven’t automated away all AI research.
You’ve mentioned the need for a missing update, and I think part of that missing update is that we didn’t really realize how large the entire internet was, and this gave the fuel for the very impressive LLM scaling, but this is finite, and could very plausibly not be enough for LLMs out of the current companies.
However, I’m inclined towards thinking the issue may not be as fundamental as you think it is, for the reason @abramdemski said below:
My earlier perspective, which asserted “LLMs are fundamentally less data-efficient than humans, because the representational capabilities of Transformers aren’t adequate for human concepts, so LLMs have to memorize many cases where humans can use one generalization” would have predicted that it is not possible to achieve GPT2 levels of linguistic competence on so little data.
Given the budgets involved, I think it is not at all surprising that only a GPT2 level of competence was reached. It therefore becomes plausible that a scaled-up effort of the same sort could reach GPT4 levels or higher with human-scale data.
The point being: it seems to me like LLMs can have similar data-efficiency to humans if effort is put in that direction. The reason we are seeing such a drastic difference now is due more to where the low-hanging fruit lies, rather than fundamental limitations of LLMs.
Remember, this is a small scale experiment, and you often have to go big in order to make use of your new findings, even if there are enough efficiency tricks such that at the end, you can make an AI that is both very capable and more efficient than modern human learning (I’m not assuming that there exists a method such that LLMs can be made more data efficient than a human, but am claiming that if they exist, there still would need to be scaling to find those efficiency tricks).
So it being only as good as GPT2 is unsurprising. Keep in mind that GPT-3 was trained by OpenAI who absolutely believed in the ability to scale up compute, and had more resources than academic groups at the time.
For example, sometimes people believe that, for some X, we just need X to make AGI from current ML systems. Sometimes they believe this because they are imputing the ghost in the machine. E.g.: “LLMs don’t get feedback from the environment, where they get to try an experiment and then see the results from the external world. When they do, they’ll be able to learn unboundedly and be fully generally intelligent.”. I think what this person is doing is imagining themselves without feedback loops with external reality; then imagining themselves with feedback loops; noticing the difference in their own thinking in those two hypotheticals; and then imputing the difference to the LLM+feedback system, imagining that the step LLM⟶ LLM+feedback is like the step human⟶ human+feedback. In this case imputing the ghost is a mistake in both ways: they don’t realize that they’re making that imputation, and the LLM+feedback system actually doesn’t have the imputed capabilities. They’re falsely imputing [all those aspects of their mind that would be turned on by going from no-feedback to yes-feedback] to the LLM+feedback. That’s a mistake because really the capabilities that come online in the human⟶ human+feedback step require a bunch of machinery that the human does have, in the background, but that the LLM doesn’t have (and the [LLM+feedback + training apparatus] system doesn’t have the machinery that [human + humanity + human evolution] has).
My main response is that once we condition on LLMs not having weight level continual learning as well as them not having a long-term memory, there’s little mystery left to explain for LLM capabilities, so there’s no other machinery that I’ve missed that is very important.
For the continual learning point, a great example of this is that humans don’t hit walls of capability nearly as often as LLMs do, and in particular human success curves on task often flatline or increase, rather than hit hard limits, and in particular when needed have very, very high conceptual resolution, such that we can work on long, open-ended problems without being entirely unproductive of insights.
And this is because human neurons constantly update, and there’s no deployment phase where all your neurons stop updating.
Human neuroplasticity declines, but never is completely gone as you age.
I explain more about why I think continual learning is important below, and @gwern really explained this far better than I can, so read Gwern’s comment too:
For the long-term memory point, the reason why it’s important for human learning is that it simultaneously prevents us from being stuck on unproductive loops like how Claude can go to the Spiritual Bliss attractor or how Claude has a very bad habit of being stuck in loops when trying to win the game of Pokemon, and also allows you to build on previous wins/take in large amounts of context without being lost, which is a key part of doing jobs.
Dwarkesh Patel explains better than I can why a lack of long-term memory/continual learning is such a big deal for LLMs, and reduces their ability to be creative, because they cannot build upon hard-earned optimizations into something bigger, and I tend to model humans getting insights not as you thinking hard for a day and fully forming the insight like Athena out of Zeus, but rather humans getting a first small win, and because they can rely on their long-term memory, they don’t have to worry about losing that first small win/insight, and they continuously look both at reality and theory to iteratively refine their insights until they finally have a big insight that comes out after a lot of build-up, but LLMs can’t ever build up to big insights, because they keep constantly forgetting the small stuff that they have gotten:
LLMs actually do get kinda smart and useful in the middle of a session. For example, sometimes I’ll co-write an essay with an LLM. I’ll give it an outline, and I’ll ask it to draft the essay passage by passage. All its suggestions up till 4 paragraphs in will be bad. So I’ll just rewrite the whole paragraph from scratch and tell it, “Hey, your shit sucked. This is what I wrote instead.” At that point, it can actually start giving good suggestions for the next paragraph. But this whole subtle understanding of my preferences and style is lost by the end of the session.
Maybe the easy solution to this looks like a long rolling context window, like Claude Code has, which compacts the session memory into a summary every 30 minutes. I just think that titrating all this rich tacit experience into a text summary will be brittle in domains outside of software engineering (which is very text-based). Again, think about the example of trying to teach someone how to play the saxophone using a long text summary of your learnings. Even Claude Code will often reverse a hard-earned optimization that we engineered together before I hit /compact—because the explanation for why it was made didn’t make it into the summary.
Edit: And the cases where there are fully formed ideas from what seems like to be nothing is because of your default mode network in the brain, and more generally you always are computing and thinking in the background, and it’s basically continual learning on your own thoughts, and once we realize this, it’s much less surprising that we can somewhat reliably create insights. LLMs lack any equivalent of a default mode network/continual learning on their own thoughts, which pretty neatly explains why people report that they have insights/creativity out of nowhere, but LLMs so far haven’t done this yet:
Another edit: A key part of my worldview is that by the 2030s, we have enough compute such that we can constantly experiment with human-brain sized architectures, and in particular given that I think capabilities are learned within life-time for various reasons @Steven Byrnes and @Quintin Pope already said, and this means that the remaining missing paradigms are likely to be discovered more quickly, and critically this doesn’t depend on LLMs becoming AGI:
An important study here that’s quoted for future reference:
October 2024: I found a paper Blumberg & Adolph 2023, which discusses the extent to which the cortex is involved in newborn behavior. Their answer is “very little” (which supports my hypothesis). I added it as a reference in Section 2.5.2.
A key crux that I hold, relative to you is that I think LLMs are in fact a little bit creative/can sometimes form insights (though with caveats), but that this is not the relevant question to be asking, and I think most LLM incapacities are not literally that they can never do this fundamentally, but rather that at realistic amounts of compute and data, they cannot reliably form insights/be creative on their own, or even do as well as the best human scientists, similar to @Thane Ruthenis’s comment below:
So the lack of long-term memory and continual learning is closer to the only bottleneck for LLMs (and I’m willing to concede that any AI that solves these bottlenecks is not a pure LLM).
Also, this part is something that I agree with, but I expect normal iteration/engineering to solve these sorts of problems reliably, so I don’t consider it a fundamental reason not to expect AGI in say the 2030s:
Most instances of a category are not the most powerful, most general instances of that category. So just because we have, or will soon have, some useful instances of a category, doesn’t strongly imply that we can or will soon be able to harness most of the power of stuff in that category.
Remember, this is a small scale experiment, and you often have to go big in order to make use of your new findings,
Ok… but are you updating on hypothetical / fictional evidence? BTW to clarify, the whole sample efficiency thing is kind of a sideline to me. If someone got GPT4 level performance by training on human data that is like 10x the size of the books that a well-read human would read in 50 years, that would be really really weird and confusing to me, and would probably shorten my timelines somewhat; in contrast, what would really shorten my timelines would be observations of LP2X creating novel interesting concepts (or at least originary interesting concepts, as in Hänni’s “Cantor’s Diagonal from scratch” thing).
My main response is that once we condition on LLMs not having weight level continual learning as well as them not having a long-term memory, there’s little mystery left to explain for LLM capabilities, so there’s no other machinery that I’ve missed that is very important.
Yes, this is a good example of mysteriously assuming for no reason that “if we just get X, then I don’t see what’s stopping my learning program from being an AGI, so therefore it is an AGI”, which makes absolutely zero sense and you should stop.
And this is because human neurons constantly update, and there’s no deployment phase where all your neurons stop updating.
No it’s not. I mean it is a little bit. But it’s also “because” “neurons implement Bayesian learning”, and it’s also “because” “neurons implement a Turing machine”. Going from this sort of “because” to “because my thing is also a Turing machine and therefore it’s smart, just like neurons, which are also Turing machines” makes zero sense.
relative to you is that I think LLMs are in fact a little bit creative/can sometimes form insights (though with caveats)
What considerations (observations, arguments, etc.) most strongly contributed to convincing you of the strongest form of this proposition that you believe?
I do not think that Noosphere’s comment did not contain an argument. The rest of the comment after the passage you cited tries to lay out a model for why continual learning and long-term memory might be the only remaining bottlenecks. Perhaps you think that this argument is very bad, but it is an argument, and I did not think that your reply to it was helpful for the discussion.
I have a couple things to add here to the conversation that I think will help:
I think a lot of current LLM failures that @TsviBT is pointing to are downstream of 2 big issues currently, which is that current LLM weights are frozen, meaning that once they stop training, they have no way to learn anything new other than ICL, and while there’s some generalization ability, it’s also not enough for real use cases like automating AI research, and in particular LLMs have no long-term memory, meaning they have to relearn things that a human would only have to learn once, and the benchmark of Claude Plays Pokemon is a really good example of what happens when you don’t have long-term memory, and that’s due to LLMs being based on transformers rather than RNNs, where hidden states allow a form of long-term memory to occur.
The non-straw version of LLMs not being able to automate jobs from humans is that context windows don’t have to scale the way they have since 2019, because DRAM isn’t scaling exponentially, and in-context learning in practice turning out to be too weak to really replace weight-level continual learning, and their lack of memory and agency meaning that they constantly get stuck in loops, meaning they are useless for the vast majority of jobs.
I agree with this statement specifically here, and a reason for this is that I believe most human capabilities don’t come from evolution, but from within life-time learning, and the areas where evolution matters are unfortunately disproportionately AI alignment-relevent areas of the human brain, which is “(1) just one or two major shifts in training technique to be enough to get us from where we are to AI research assistants capable of major innovation”.
An independent reason to think AGI will come in the 2030s most likely, assuming LLMs do plateau, is because we are getting into the regime where we can more inexpensively experiment with human-brain level compute, and we are scaling through a lot of OOMs for compute usage, meaning that we can inexpensively experiment for insights a lot more than we currently do:
https://www.lesswrong.com/posts/yew6zFWAKG4AGs3Wk/?commentId=Bu8qnHcdsv4szFib7
And to answer @Cole Wyeth’s question of why one human brain lifetime and not evolutionary timelines, the short answer is most of the capabilities are rederived for humans every lifetime, and evolution matters way less than we think it does.
@Steven Byrnes and @Quintin Pope have discussed this before:
https://www.lesswrong.com/posts/wAczufCpMdaamF9fy/my-objections-to-we-re-all-gonna-die-with-eliezer-yudkowsky#Edit__Why_evolution_is_not_like_AI_training
https://www.lesswrong.com/posts/wBHSYwqssBGCnwvHg/intro-to-brain-like-agi-safety-2-learning-from-scratch-in
https://www.lesswrong.com/posts/hvz9qjWyv8cLX9JJR/evolution-provides-no-evidence-for-the-sharp-left-turn#Evolution_s_sharp_left_turn_happened_for_evolution_specific_reasons
https://www.lesswrong.com/posts/pz7Mxyr7Ac43tWMaC/against-evolution-as-an-analogy-for-how-humans-will-create
I agree with Cole Wyeth that current LLMs are pretty bad at agency, and if we assume they don’t scale to better agency over time, I’d put much lower probability on LLMs being able to automate away the remaining bottlenecks to ASI, and it’s a reasonable hypothesis to hold (I’d put about 52% probability on this currently)
And in particular, I think the fact that LLM capability degrades way faster than humans as @abramdemski saw is tied to a lack of continual learning, and ICL not being enough currently to actually subsititute for weight-level continual learning.
And yet, I think there are good reasons to believe independently of LLMs that AGI/ASI is coming pretty soon, and I do think there’s reason to believe that timelines are probably short, even if LLMs do plateau.
To be clear, I think this is worse than a future where LLMs do just straight up scale to ASI.
Also, the entire crux is basically “does their in-context learning/generalization and creativity shown in current LLMs actually act as a proper subsitute for continual learning in weights and memory?”
The other crux is whether context windows will actually continue to scale in the way they have since 2019.
I don’t agree with this connection. Why would you think that continual learning would help with this specific sort of thing? It seems relevantly similar to just throwing more training data at the problem, which has shown only modest progress so far.
The key reason is to bend the shape of the curve, and my key crux is I don’t expect throwing more training data to change the shape of the curve where past a certain point, LLMs sigmoid/fall off hard, and my expectation is more training data would make LLMs improve, but they’d still have a point where once LLMs are asked to do any task harder than that point, LLMs start becoming incapable more rapidly in humans.
To quote Gwern:
From this link:
https://www.lesswrong.com/posts/deesrjitvXM4xYGZd/metr-measuring-ai-ability-to-complete-long-tasks#hSkQG2N8rkKXosLEF
(Note that I have a limit on how many comments I can make per week, so I will likely respond slowly, if at all to any responses to this).
It seems to me like the improvement in learning needed for what Gwern describes has little to do with “continual” and is more like “better learning” (better generalization, generalization from less examples).
Regarding continual learning and memory, I mentioned in the dialogue that I’m not just talking about performance of trained LLMs, but rather addressing the whole Architecture:
Your remarks sound to me like “We just need X”, which I addressed here: https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#_We_just_need_X__intuitions
See also https://tsvibt.blogspot.com/2023/09/a-hermeneutic-net-for-agency.html#silently-imputing-the-ghost-in-the-machine , which I’ll quote from:
Note that I was talking about both long-term memory and continual learning, not just continual learning, so I’m happy to concede that my proposed architecture is not like how LLMs are trained today, and thus could reasonably be called a non-LLM architecture.
Though I will say that the BabyLM challenge and to a lesser extent the connect the dots paper is evidence that part of the reason current LLMs are so data inefficient is not because of fundamental limitations, but rather because AI companies didn’t really need to have LLMs be data efficient in order for LLMs to work so far, but by 2028-2030, this won’t work nearly as effectively assuming LLMs haven’t automated away all AI research.
You’ve mentioned the need for a missing update, and I think part of that missing update is that we didn’t really realize how large the entire internet was, and this gave the fuel for the very impressive LLM scaling, but this is finite, and could very plausibly not be enough for LLMs out of the current companies.
However, I’m inclined towards thinking the issue may not be as fundamental as you think it is, for the reason @abramdemski said below:
Remember, this is a small scale experiment, and you often have to go big in order to make use of your new findings, even if there are enough efficiency tricks such that at the end, you can make an AI that is both very capable and more efficient than modern human learning (I’m not assuming that there exists a method such that LLMs can be made more data efficient than a human, but am claiming that if they exist, there still would need to be scaling to find those efficiency tricks).
So it being only as good as GPT2 is unsurprising. Keep in mind that GPT-3 was trained by OpenAI who absolutely believed in the ability to scale up compute, and had more resources than academic groups at the time.
To respond to this:
My main response is that once we condition on LLMs not having weight level continual learning as well as them not having a long-term memory, there’s little mystery left to explain for LLM capabilities, so there’s no other machinery that I’ve missed that is very important.
For the continual learning point, a great example of this is that humans don’t hit walls of capability nearly as often as LLMs do, and in particular human success curves on task often flatline or increase, rather than hit hard limits, and in particular when needed have very, very high conceptual resolution, such that we can work on long, open-ended problems without being entirely unproductive of insights.
And this is because human neurons constantly update, and there’s no deployment phase where all your neurons stop updating.
Human neuroplasticity declines, but never is completely gone as you age.
I explain more about why I think continual learning is important below, and @gwern really explained this far better than I can, so read Gwern’s comment too:
https://www.lesswrong.com/posts/5tqFT3bcTekvico4d/do-confident-short-timelines-make-sense#mibF9KKtuJxtnDBne
For the long-term memory point, the reason why it’s important for human learning is that it simultaneously prevents us from being stuck on unproductive loops like how Claude can go to the Spiritual Bliss attractor or how Claude has a very bad habit of being stuck in loops when trying to win the game of Pokemon, and also allows you to build on previous wins/take in large amounts of context without being lost, which is a key part of doing jobs.
Dwarkesh Patel explains better than I can why a lack of long-term memory/continual learning is such a big deal for LLMs, and reduces their ability to be creative, because they cannot build upon hard-earned optimizations into something bigger, and I tend to model humans getting insights not as you thinking hard for a day and fully forming the insight like Athena out of Zeus, but rather humans getting a first small win, and because they can rely on their long-term memory, they don’t have to worry about losing that first small win/insight, and they continuously look both at reality and theory to iteratively refine their insights until they finally have a big insight that comes out after a lot of build-up, but LLMs can’t ever build up to big insights, because they keep constantly forgetting the small stuff that they have gotten:
https://www.dwarkesh.com/p/timelines-june-2025
Edit: And the cases where there are fully formed ideas from what seems like to be nothing is because of your default mode network in the brain, and more generally you always are computing and thinking in the background, and it’s basically continual learning on your own thoughts, and once we realize this, it’s much less surprising that we can somewhat reliably create insights. LLMs lack any equivalent of a default mode network/continual learning on their own thoughts, which pretty neatly explains why people report that they have insights/creativity out of nowhere, but LLMs so far haven’t done this yet:
https://gwern.net/ai-daydreaming#continual-thinking
Another edit: A key part of my worldview is that by the 2030s, we have enough compute such that we can constantly experiment with human-brain sized architectures, and in particular given that I think capabilities are learned within life-time for various reasons @Steven Byrnes and @Quintin Pope already said, and this means that the remaining missing paradigms are likely to be discovered more quickly, and critically this doesn’t depend on LLMs becoming AGI:
https://www.lesswrong.com/posts/yew6zFWAKG4AGs3Wk/?commentId=Bu8qnHcdsv4szFib7
An important study here that’s quoted for future reference:
A key crux that I hold, relative to you is that I think LLMs are in fact a little bit creative/can sometimes form insights (though with caveats), but that this is not the relevant question to be asking, and I think most LLM incapacities are not literally that they can never do this fundamentally, but rather that at realistic amounts of compute and data, they cannot reliably form insights/be creative on their own, or even do as well as the best human scientists, similar to @Thane Ruthenis’s comment below:
https://www.lesswrong.com/posts/GADJFwHzNZKg2Ndti/have-llms-generated-novel-insights#YFfperxLEnWoomzrH
So the lack of long-term memory and continual learning is closer to the only bottleneck for LLMs (and I’m willing to concede that any AI that solves these bottlenecks is not a pure LLM).
Also, this part is something that I agree with, but I expect normal iteration/engineering to solve these sorts of problems reliably, so I don’t consider it a fundamental reason not to expect AGI in say the 2030s:
Ok… but are you updating on hypothetical / fictional evidence? BTW to clarify, the whole sample efficiency thing is kind of a sideline to me. If someone got GPT4 level performance by training on human data that is like 10x the size of the books that a well-read human would read in 50 years, that would be really really weird and confusing to me, and would probably shorten my timelines somewhat; in contrast, what would really shorten my timelines would be observations of LP2X creating novel interesting concepts (or at least originary interesting concepts, as in Hänni’s “Cantor’s Diagonal from scratch” thing).
Yes, this is a good example of mysteriously assuming for no reason that “if we just get X, then I don’t see what’s stopping my learning program from being an AGI, so therefore it is an AGI”, which makes absolutely zero sense and you should stop.
No it’s not. I mean it is a little bit. But it’s also “because” “neurons implement Bayesian learning”, and it’s also “because” “neurons implement a Turing machine”. Going from this sort of “because” to “because my thing is also a Turing machine and therefore it’s smart, just like neurons, which are also Turing machines” makes zero sense.
What considerations (observations, arguments, etc.) most strongly contributed to convincing you of the strongest form of this proposition that you believe?
@Lucius Bushnaq It’s not too combative, you’re wrong. My previous comment laid out what’s wrong with the reasoning. Then Noosphere89 wrote a big long comment that makes all the same lines of reasoning, still without giving any arguments. This is really bad epistemics, and people going around vibing hard about this have been poisoning (or rather, hijacking https://www.lesswrong.com/posts/dAz45ggdbeudKAXiF/a-regime-change-power-vacuum-conjecture-about-group-belief) the discourse for 5 years.
I do not think that Noosphere’s comment did not contain an argument. The rest of the comment after the passage you cited tries to lay out a model for why continual learning and long-term memory might be the only remaining bottlenecks. Perhaps you think that this argument is very bad, but it is an argument, and I did not think that your reply to it was helpful for the discussion.