There are many missing cognitive faculties, whose absence can plausibly be compensated with scale and other AI advantages. We haven’t yet run out of scale, though in 2030s we will (absent AGI, trillions of dollars in revenue).
The currently visible crucial things that are missing are sample efficiency (research taste) and continual learning, with many almost-ready techniques to help with the latter. Sholto Douglas of Anthropic claimed a few days ago that probably continual learning gets solved in a satisfying way in 2026 (at 38:29 in the podcast). Dario Amodei previously discussed how even in-context learning might confer the benefits of continual learning with further scaling and sufficiently long contexts (at 13:17 in the podcast). Dwarkesh Patel says there are rumors Sutskever’s SSI is working on test time training (at 39:25 in the podcast). Thinking Machines published work on better LoRA, which in some form seems crucial to making continual learning via weight updates practical for individual model instances. This indirectly suggests OpenAI would also have a current major project around continual learning.
The recent success of RLVR suggests that any given sufficiently narrow mode of activity (such as doing well on a given kind of benchmarks) can now be automated. This plausibly applies to RLVR itself, which might be used by AIs to “manually” add new skills to themselves, after RLVR was used to teach AIs to apply RLVR to themselves, in the schleppy way that AI researchers currently do to get them better at benchmarkable activities. AI instances doing this automatically for the situation (goal, source of tasks, job) where they find themselves covers a lot of what continual learning is supposed to do. Sholto Douglas again (at 1:00:54 in a recent podcast):
So far the evidence indicates that our current methods haven’t yet found a problem domain that isn’t tractable with sufficient effort.
So it’s not completely clear that there are any non-obvious obstacles still remaining. Missing research taste might get paved over with sufficient scale of effort, once continual learning ensures there is some sustained progress at all when AIs are let loose to self-improve. To know that some obstacles are real, the field first needs to run out of scaling and have a few years to apply RLVR (develop RL environments) to automate all the obvious things that might help AIs “manually” compensate for the missing faculties.
I have long respected your voice on this website, and I appreciate you chiming in with a lot of tactical, practical, well-cited points about the degree to which “seed AI” already may exist in a qualitative way whose improvement/cost ratio or improvement/time ratio isn’t super high yet, but might truly exist already (and hence “AGI” in the new modern goal-shifted sense might exist already (and hence the proximity of “ASI” in the new modern goal-shifted sense might simply be “a certain budget away” rather than a certain number of months or years)).
A deep part of my sadness about the way that the terminology for this stuff is so fucky is how the fuckiness obscures the underlying reality from many human minds who might otherwise orient to things in useful ways and respond with greater fluidity.
If names be not correct, language is not in accordance with the truth of things. If language be not in accordance with the truth of things, affairs cannot be carried on to success. When affairs cannot be carried on to success, proprieties and music do not flourish. When proprieties and music do not flourish, punishments will not be properly awarded. When punishments are not properly awarded, the people do not know how to move hand or foot. Therefore a superior man considers it necessary that the names he uses may be spoken appropriately, and also that what he speaks may be carried out appropriately. What the superior man requires is just that in his words there may be nothing incorrect.
Back in 2023 when GPT-4 was launched, a few people took its capabilities as a sign that superintelligence might be a year or two away. It seems to me that the next clear qualitative leap came with chain-of-thought and reasoning models, starting with OpenAI’s o-series. Would you agree with that?
There are many missing cognitive faculties, whose absence can plausibly be compensated with scale and other AI advantages. We haven’t yet run out of scale, though in 2030s we will (absent AGI, trillions of dollars in revenue).
The currently visible crucial things that are missing are sample efficiency (research taste) and continual learning, with many almost-ready techniques to help with the latter. Sholto Douglas of Anthropic claimed a few days ago that probably continual learning gets solved in a satisfying way in 2026 (at 38:29 in the podcast). Dario Amodei previously discussed how even in-context learning might confer the benefits of continual learning with further scaling and sufficiently long contexts (at 13:17 in the podcast). Dwarkesh Patel says there are rumors Sutskever’s SSI is working on test time training (at 39:25 in the podcast). Thinking Machines published work on better LoRA, which in some form seems crucial to making continual learning via weight updates practical for individual model instances. This indirectly suggests OpenAI would also have a current major project around continual learning.
The recent success of RLVR suggests that any given sufficiently narrow mode of activity (such as doing well on a given kind of benchmarks) can now be automated. This plausibly applies to RLVR itself, which might be used by AIs to “manually” add new skills to themselves, after RLVR was used to teach AIs to apply RLVR to themselves, in the schleppy way that AI researchers currently do to get them better at benchmarkable activities. AI instances doing this automatically for the situation (goal, source of tasks, job) where they find themselves covers a lot of what continual learning is supposed to do. Sholto Douglas again (at 1:00:54 in a recent podcast):
So it’s not completely clear that there are any non-obvious obstacles still remaining. Missing research taste might get paved over with sufficient scale of effort, once continual learning ensures there is some sustained progress at all when AIs are let loose to self-improve. To know that some obstacles are real, the field first needs to run out of scaling and have a few years to apply RLVR (develop RL environments) to automate all the obvious things that might help AIs “manually” compensate for the missing faculties.
I have long respected your voice on this website, and I appreciate you chiming in with a lot of tactical, practical, well-cited points about the degree to which “seed AI” already may exist in a qualitative way whose improvement/cost ratio or improvement/time ratio isn’t super high yet, but might truly exist already (and hence “AGI” in the new modern goal-shifted sense might exist already (and hence the proximity of “ASI” in the new modern goal-shifted sense might simply be “a certain budget away” rather than a certain number of months or years)).
A deep part of my sadness about the way that the terminology for this stuff is so fucky is how the fuckiness obscures the underlying reality from many human minds who might otherwise orient to things in useful ways and respond with greater fluidity.
As was said in days of yore:
Back in 2023 when GPT-4 was launched, a few people took its capabilities as a sign that superintelligence might be a year or two away. It seems to me that the next clear qualitative leap came with chain-of-thought and reasoning models, starting with OpenAI’s o-series. Would you agree with that?