Can you expand on this, and in particular, how it gets you to be quite confident (relative to the prior where any given innovation isn’t confidently going to grow into superintelligence)?
This question seems to be insisting on a weird burden of proof, to me. I’m going to try to answer it straightforwardly, but I imagine my answers will feel frustrating, or missing the point, or something.
The AI agents we have now meet many to most of the criteria for AGI that many people put forward in previous decades. It’s not crazy to say that Claude 4.6 is an AGI, and insofar as it isn’t, its not very clear what’s missing.
Last I checked, GPT-5.something was the 6th best competitive programmer in the world. The AIs are winning gold in math Olympiad competitions. They’re clearly better than me, already, at almost all of technical thinking.
I don’t see a fundamental reason they shouldn’t be able to eg design components of a nuclear reactor, or design a whole nuclear reactor, better than than the best human nuclear engineers, except that their time horizon is too short to make much progress. But the AI time horizons have been doubling on a consistent exponential since GPT-3 at least.
And notably, on the path to getting here, we went from language models that mostly output gibberish, to chatbots that are vastly more knowledgeable than any human, to agents that can solve open profesional level math problem and build software better than most humans and faster than almost every human, by applying very simple and obvious ideas.
“Huh, GPT-2 worked pretty well. What if we do the same thing, with a bigger model, on more data”. This worked.
“What if we have the model train on procedurally generated math and programming problems”. This worked.
“What if we build simple scaffold that queries the model, to implement an agent.” This didn’t work at first, but a combination of making the models better, making better scaffolding, and training the model in the scaffolding got it to work.
I am not an AI researcher. But I had all of these ideas, years before the AI companies got them working. (In contrast, I would need a lot of technical expertise, and maybe more raw intelligence than I have, to invent eg the attention mechanism.)
The amazing progress in AI over the past 5 years has been driven by taking very obvious ideas, and working out the engineering details to implement them.
I don’t have any strong reason to think that this trend of applying basically simple ideas to LLMs and getting increasingly impressive capabilities will break. There were lots and lots of things LLMs couldn’t do well, that people claimed were fundamental problems, that were solved naturally in the course of scaling.
There are still some gaps left. Notably, their time horizons are too short—but as noted, progress is being made continually on that. Also, LLMs can’t really invent new concepts, at least after training, which seems like a blocker for really doing science. Humans are still vastly more sample-efficient than AIs in training.
Do I know that all the gaps that are left in the LLM agents will be solved by the continual application of basically simple ideas and engineering schlep? No, obviously not. But I also don’t have any strong reason to think that they won’t be.
Eliezer was saying in 2021 that GPTs are only memorizing “shallow patterns” and so don’t embody the deep parts of cognition. It’s unclear if that’s a correct gloss. If it is true, it turns out you can get surprisingly technically competent by only memorizing and applying shallow patterns. It turns out the linguistic traces of human thought have a lot more of the true generators of human thought contained within them, than I would have guessed.
And in any case, we’re doing RLVF now, and so the AIs can bootstrap from human concepts to learn from their own trial and error, just like alpha-go. Maybe you can get all the way to AIs that are superhuman in ~every domain, by just designing a huge number of RL environments, and giving the agents the affordance to design their own RL environments to train on in response to novel circumstances, despite still having weak fluid intelligence.
Plus there are more-or-less obvious ideas for continual learning which seem like they would enable the AIs to develop new concepts.
Given all this, it feels like at least 30%(?) that everything left to do can be automated by the LLM agents of the next two years.
Again, modus ponens meet modus tollens and Amdahl. According to your argument, the Industrial Revolution (or the invention of computers, or the invention of compilers, or operating systems, or the internet, or Google) should imminently create AGI because it frees up a bunch of human capital. Now, that’s probably true! Just not on any specific timeline.
Well the difference in this case is how close we already are (or seem to be). When the industrial revolution happened, a or when computers or compilers were invented, there were still many scientific discoveries to be made and engineering challenges to solve, between us and AGI.
Now, there are plausibly only engineering challenges, which we can automate with our existing AIs, and if not, probably only one or two scientific discoveries left.
I think it should be obviously crazy to someone who basically just updated off of a bunch of other people updating
This is unimportant, but I find the insinuation that everyone is updating on everyone else’s views kind of annoying. I’m not immune to the hype cycles, but I feel like I’m mostly updating on things that I’m seeing with my own eyes. It’s fine though—I can imagine how gaslighting it might feel if almost everyone around me was asserting or assuming some very important point, and none of them could manage to make an argument for that point.
This question seems to be insisting on a weird burden of proof, to me.
I’m not sure what’s weird about it, but yes, I think someone claiming to predict the future confidently as opposed to the more default background broad uncertainty would have the burden of proof.
The AI agents we have now meet many to most of the criteria for AGI that many people put forward in previous decades.
Do you think it has high fluid intelligence (assuming as best you can, arguendo, that this phrase maps to something meaningful + important)? If yes, why (given that you’d be disagreeing with a lot of other short timelines views)? If no, why talk about AGI that doesn’t include high fluid intelligence?
and insofar as it isn’t, its not very clear what’s missing.
I don’t see a fundamental reason they shouldn’t be able to
Do I know that all the gaps that are left in the LLM agents will be solved by the continual application of basically simple ideas and engineering schlep? No, obviously not. But I also don’t have any strong reason to think that they won’t be.
I think it’s true and good to be worried that the AI research community could adaptive creatively surprise us with inventing AGI seedstuff in 1 year or 5 years. What I’m arguing against, or just trying to understand, is people (such as you) seeming to have very high confidence in this (like having a median of 5 years, i.e. 50% chance). That sure sounds like positive knowledge of us having almost all of fluid intelligence AGI seedstuff. No?
They’re clearly better than me, already, at almost all of technical thinking.
Except for the most important parts, such as orienting to a new domain / new question in a manner that produces successful understanding in the long run.
I don’t have any strong reason to think that this trend of applying basically simple ideas to LLMs and getting increasingly impressive capabilities will break.
Response 1: This type of reasoning does not work for all those other previous big breakthroughs (such as the invention of the universal computer, of the operating system, or of google search).
Response 2: Consider the hypothesis that it went up fast because it used up available data. And, as you can see in the rest of the top-level thread (e.g. from Kokotaljo and Greenblatt) that in fact people use this as an excuse (as it were) for LLMs performing badly on some things.
I am not an AI researcher. But I had all of these ideas, years before the AI companies got them working.
The amazing progress in AI over the past 5 years has been driven by taking very obvious ideas, and working out the engineering details to implement them.
Which suggests that they aren’t much of an idea. Which suggests that we don’t understand much about intelligence. I think you’re trying to say “turns out that fairly obvious things are most of intelligence”. And I’m trying to say “actually most of that was just unlocking what was already fairly shallowly available in the massive training corpus, so we did not get much evidence that we understood / can build the generators of that corpus or of fluid / general intelligence in general”.
. If it is true, it turns out you can get surprisingly technically competent by only memorizing and applying shallow patterns. It turns out the linguistic traces of human thought have a lot more of the true generators of human thought contained within them, t
Wait can you expand on this? Why do you think true generators of human thought are contained in them / picked up by LLMs?
Now, there are plausibly only engineering challenges, which we can automate with our existing AIs, and if not, probably only one or two scientific discoveries left.
This question seems to be insisting on a weird burden of proof, to me. I’m going to try to answer it straightforwardly, but I imagine my answers will feel frustrating, or missing the point, or something.
The AI agents we have now meet many to most of the criteria for AGI that many people put forward in previous decades. It’s not crazy to say that Claude 4.6 is an AGI, and insofar as it isn’t, its not very clear what’s missing.
Last I checked, GPT-5.something was the 6th best competitive programmer in the world. The AIs are winning gold in math Olympiad competitions. They’re clearly better than me, already, at almost all of technical thinking.
I don’t see a fundamental reason they shouldn’t be able to eg design components of a nuclear reactor, or design a whole nuclear reactor, better than than the best human nuclear engineers, except that their time horizon is too short to make much progress. But the AI time horizons have been doubling on a consistent exponential since GPT-3 at least.
And notably, on the path to getting here, we went from language models that mostly output gibberish, to chatbots that are vastly more knowledgeable than any human, to agents that can solve open profesional level math problem and build software better than most humans and faster than almost every human, by applying very simple and obvious ideas.
“Huh, GPT-2 worked pretty well. What if we do the same thing, with a bigger model, on more data”. This worked.
“What if we have the model train on procedurally generated math and programming problems”. This worked.
“What if we build simple scaffold that queries the model, to implement an agent.” This didn’t work at first, but a combination of making the models better, making better scaffolding, and training the model in the scaffolding got it to work.
I am not an AI researcher. But I had all of these ideas, years before the AI companies got them working. (In contrast, I would need a lot of technical expertise, and maybe more raw intelligence than I have, to invent eg the attention mechanism.)
The amazing progress in AI over the past 5 years has been driven by taking very obvious ideas, and working out the engineering details to implement them.
I don’t have any strong reason to think that this trend of applying basically simple ideas to LLMs and getting increasingly impressive capabilities will break. There were lots and lots of things LLMs couldn’t do well, that people claimed were fundamental problems, that were solved naturally in the course of scaling.
There are still some gaps left. Notably, their time horizons are too short—but as noted, progress is being made continually on that. Also, LLMs can’t really invent new concepts, at least after training, which seems like a blocker for really doing science. Humans are still vastly more sample-efficient than AIs in training.
Do I know that all the gaps that are left in the LLM agents will be solved by the continual application of basically simple ideas and engineering schlep? No, obviously not. But I also don’t have any strong reason to think that they won’t be.
Eliezer was saying in 2021 that GPTs are only memorizing “shallow patterns” and so don’t embody the deep parts of cognition. It’s unclear if that’s a correct gloss. If it is true, it turns out you can get surprisingly technically competent by only memorizing and applying shallow patterns. It turns out the linguistic traces of human thought have a lot more of the true generators of human thought contained within them, than I would have guessed.
And in any case, we’re doing RLVF now, and so the AIs can bootstrap from human concepts to learn from their own trial and error, just like alpha-go. Maybe you can get all the way to AIs that are superhuman in ~every domain, by just designing a huge number of RL environments, and giving the agents the affordance to design their own RL environments to train on in response to novel circumstances, despite still having weak fluid intelligence.
Plus there are more-or-less obvious ideas for continual learning which seem like they would enable the AIs to develop new concepts.
Given all this, it feels like at least 30%(?) that everything left to do can be automated by the LLM agents of the next two years.
Well the difference in this case is how close we already are (or seem to be). When the industrial revolution happened, a or when computers or compilers were invented, there were still many scientific discoveries to be made and engineering challenges to solve, between us and AGI.
Now, there are plausibly only engineering challenges, which we can automate with our existing AIs, and if not, probably only one or two scientific discoveries left.
This is unimportant, but I find the insinuation that everyone is updating on everyone else’s views kind of annoying. I’m not immune to the hype cycles, but I feel like I’m mostly updating on things that I’m seeing with my own eyes. It’s fine though—I can imagine how gaslighting it might feel if almost everyone around me was asserting or assuming some very important point, and none of them could manage to make an argument for that point.
I’m not sure what’s weird about it, but yes, I think someone claiming to predict the future confidently as opposed to the more default background broad uncertainty would have the burden of proof.
Yes, but one has to update one’s beliefs about everything (or more feasibly, about the most relevant things), not just about one thing. There is a missing update here: those people had bad models, and in fact those tasks are shocking not only achievable through general intelligence. See https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#Things_that_might_actually_work:~:text=There is a-,missing update,-. We see impressive
Do you think it has high fluid intelligence (assuming as best you can, arguendo, that this phrase maps to something meaningful + important)? If yes, why (given that you’d be disagreeing with a lot of other short timelines views)? If no, why talk about AGI that doesn’t include high fluid intelligence?
Just because I, a non-engineer, cannot explain to you in detail why this pile of steel beams will not stand up as a bridge, does not mean it is a bridge. See https://www.lesswrong.com/posts/sTDfraZab47KiRMmT/views-on-when-agi-comes-and-on-strategy-to-reduce#The__no_blockers__intuition
I think it’s true and good to be worried that the AI research community could adaptive creatively surprise us with inventing AGI seedstuff in 1 year or 5 years. What I’m arguing against, or just trying to understand, is people (such as you) seeming to have very high confidence in this (like having a median of 5 years, i.e. 50% chance). That sure sounds like positive knowledge of us having almost all of fluid intelligence AGI seedstuff. No?
Except for the most important parts, such as orienting to a new domain / new question in a manner that produces successful understanding in the long run.
Response 1: This type of reasoning does not work for all those other previous big breakthroughs (such as the invention of the universal computer, of the operating system, or of google search).
Response 2: Consider the hypothesis that it went up fast because it used up available data. And, as you can see in the rest of the top-level thread (e.g. from Kokotaljo and Greenblatt) that in fact people use this as an excuse (as it were) for LLMs performing badly on some things.
Which suggests that they aren’t much of an idea. Which suggests that we don’t understand much about intelligence. I think you’re trying to say “turns out that fairly obvious things are most of intelligence”. And I’m trying to say “actually most of that was just unlocking what was already fairly shallowly available in the massive training corpus, so we did not get much evidence that we understood / can build the generators of that corpus or of fluid / general intelligence in general”.
Wait can you expand on this? Why do you think true generators of human thought are contained in them / picked up by LLMs?
What makes you think we’re close?