The red pill is that even humans are not an upper bound for how hard this can be, that even a fully human equivalent AI doesn’t yet close an RSI loop that goes FOOM, that it would still take a lot of time after that, even when humans no longer have anything to contribute. This is actually a popular view, for people who say AGI remains a normal technology and just keeps scaling the economy, with maybe 20% growth per year rather than a doubling of industry every few days until the Sun is eaten, with a Sun-scale amount of probes soon en route to distant galaxies.
On the other hand, evolution doesn’t have a mind, so reaching even the human level is not necessary to close a loop that goes on to automatically reach human capabilities and then goes further, the only question is speed and feasibility. I think automated sample efficient learning (that adapts to any consideration that comes up) is plausibly the last piece, with RLVR already sample efficient (with respect to the data defining tasks) and able to do the cognitive heavy lifting, and pretraining already able to form a coherent picture of everything that’s been discovered so far.
Automation of routine AI R&D (merely carrying out all the plumbing of training data preparation and model training that is currently done by humans at AI companies, rather than inventing anything new at the object level of this process) is plausibly a straightforward way of getting there. RLVR-trained agents will plausibly be able to manage this soon, and it’s no doubt being explicitly attempted at every step where it’s feasible to attempt. The Sonnet-Opus-Mythos story suggests that the 100T param models of 2028-2029 might well suffice to manage every single routine step.
There still remains the possibility that the advancements from a closed loop initially remain slow. I expect cognitive self-sufficiency for AI quickly generates a hoard of conceptual inventions, something like the scientific literature that’s generated much faster, which pushes through any remaining hobblings that would otherwise promise to keep some of the other things slow for a while.
Interesting points… I don’t think it’s right to say that RLVR does all or even most of the cognitive heavy lifting; it does some of it but not other of it. I agree with your “plausibly”s, but we might put pretty different probabilities, IDK.
My suspicion would be that human-level (in the relevant dimensions) actually is special.
The chimp-human boundary goes from useless for going faster than evolution to eminently useful. But LLMs can talk and solve IMO problems, while chimps can’t, so I wouldn’t count on LLMs not already being beyond this boundary. LLMs merely need to somehow become an engine of a closed loop that works towards stronger cognitive capabilities, without necessarily themselves possessing such capabilities, or even broad human-level capabilities. Evolution is too slow to usefully do this within modern compute, but some LLM-juggling process could be much faster. And humans, when not part of the closed loop of human culture and civilization, remain as useless as chimps in reaching for superintelligence.
(RLVR is clearly deficient in the jaggedness of its results in practice, but that’s plausibly a problem of RLVR training data not being bitter-pilled. And conceptual invention might need many steps of using RLVR-trained reasoning to formulate new RLVR tasks for training the next step. So automation of generation of training data for RLVR, and of its application in training, might compensate for these issues well enough.)
The red pill is that even humans are not an upper bound for how hard this can be, that even a fully human equivalent AI doesn’t yet close an RSI loop that goes FOOM, that it would still take a lot of time after that, even when humans no longer have anything to contribute. This is actually a popular view, for people who say AGI remains a normal technology and just keeps scaling the economy, with maybe 20% growth per year rather than a doubling of industry every few days until the Sun is eaten, with a Sun-scale amount of probes soon en route to distant galaxies.
On the other hand, evolution doesn’t have a mind, so reaching even the human level is not necessary to close a loop that goes on to automatically reach human capabilities and then goes further, the only question is speed and feasibility. I think automated sample efficient learning (that adapts to any consideration that comes up) is plausibly the last piece, with RLVR already sample efficient (with respect to the data defining tasks) and able to do the cognitive heavy lifting, and pretraining already able to form a coherent picture of everything that’s been discovered so far.
Automation of routine AI R&D (merely carrying out all the plumbing of training data preparation and model training that is currently done by humans at AI companies, rather than inventing anything new at the object level of this process) is plausibly a straightforward way of getting there. RLVR-trained agents will plausibly be able to manage this soon, and it’s no doubt being explicitly attempted at every step where it’s feasible to attempt. The Sonnet-Opus-Mythos story suggests that the 100T param models of 2028-2029 might well suffice to manage every single routine step.
There still remains the possibility that the advancements from a closed loop initially remain slow. I expect cognitive self-sufficiency for AI quickly generates a hoard of conceptual inventions, something like the scientific literature that’s generated much faster, which pushes through any remaining hobblings that would otherwise promise to keep some of the other things slow for a while.
Interesting points… I don’t think it’s right to say that RLVR does all or even most of the cognitive heavy lifting; it does some of it but not other of it. I agree with your “plausibly”s, but we might put pretty different probabilities, IDK.
My suspicion would be that human-level (in the relevant dimensions) actually is special.
The chimp-human boundary goes from useless for going faster than evolution to eminently useful. But LLMs can talk and solve IMO problems, while chimps can’t, so I wouldn’t count on LLMs not already being beyond this boundary. LLMs merely need to somehow become an engine of a closed loop that works towards stronger cognitive capabilities, without necessarily themselves possessing such capabilities, or even broad human-level capabilities. Evolution is too slow to usefully do this within modern compute, but some LLM-juggling process could be much faster. And humans, when not part of the closed loop of human culture and civilization, remain as useless as chimps in reaching for superintelligence.
(RLVR is clearly deficient in the jaggedness of its results in practice, but that’s plausibly a problem of RLVR training data not being bitter-pilled. And conceptual invention might need many steps of using RLVR-trained reasoning to formulate new RLVR tasks for training the next step. So automation of generation of training data for RLVR, and of its application in training, might compensate for these issues well enough.)