I think there’s currently no visibly approaching fire
My understanding of your point-of-view is that LLM-based AI doesn’t look like a danger because it lacks continuous learning and sample efficiency. Which is true, but the raw power that this kind of AI already has, when it comes to parsing technical discourse, writing it, and generating ideas, is enough for me to disagree. That core of technical intelligence may already be enough to remedy those gaps, either by designing a better successor or just by designing new scaffolding for itself.
your point-of-view … LLM-based AI … lacks continuous learning and sample efficiency
Sample efficiency is not a problem with RLVR, LLMs can reach or exceed world class human capability (at any given sufficiently specific thing) with a reasonable amount of training data in the form of tasks/environments/graders. And this will probably get automated (within the remaining runway of raw scaling for LLMs), with slow prosaic RSI of LLMs automatically preparing such training data and using it to train the next model. Continual learning in the sense of unlimited context is not helpful because in-context learning doesn’t train deep skills (LLMs can’t learn to play chess well by looking at memory notes in their context).
The crucial constraint is that retraining with RLVR in the current paradigm (by preparing the next model) happens too slowly, much slower than token generation. Amdahl’s law then destroys the advantage of 100x faster idea generation and problem solving (using the current skills and understanding) in the overall task of research, because the subtask of learning novel deep skills doesn’t have a speedup compared to humans (and in fact might happen much slower). Raw intelligence doesn’t get around the lack of a speedup, because LLMs won’t get to vastly superhuman levels within the remaining runway of scaling in the next few years.
So continual learning in the sense of doing something like RLVR on the fly would indeed solve this hobbling, but it’s currently an unsolved problem. If LLMs can’t do conceptual research faster than humans because of their long learning loop (even if the learning becomes automated in a general way, making LLMs able to eventually learn any specific skill), and they aren’t vastly superhuman in coming up with ideas far outside their current skills and understanding, then they won’t be crucial in solving this hobbling either.
My understanding of your point-of-view is that LLM-based AI doesn’t look like a danger because it lacks continuous learning and sample efficiency. Which is true, but the raw power that this kind of AI already has, when it comes to parsing technical discourse, writing it, and generating ideas, is enough for me to disagree. That core of technical intelligence may already be enough to remedy those gaps, either by designing a better successor or just by designing new scaffolding for itself.
Sample efficiency is not a problem with RLVR, LLMs can reach or exceed world class human capability (at any given sufficiently specific thing) with a reasonable amount of training data in the form of tasks/environments/graders. And this will probably get automated (within the remaining runway of raw scaling for LLMs), with slow prosaic RSI of LLMs automatically preparing such training data and using it to train the next model. Continual learning in the sense of unlimited context is not helpful because in-context learning doesn’t train deep skills (LLMs can’t learn to play chess well by looking at memory notes in their context).
The crucial constraint is that retraining with RLVR in the current paradigm (by preparing the next model) happens too slowly, much slower than token generation. Amdahl’s law then destroys the advantage of 100x faster idea generation and problem solving (using the current skills and understanding) in the overall task of research, because the subtask of learning novel deep skills doesn’t have a speedup compared to humans (and in fact might happen much slower). Raw intelligence doesn’t get around the lack of a speedup, because LLMs won’t get to vastly superhuman levels within the remaining runway of scaling in the next few years.
So continual learning in the sense of doing something like RLVR on the fly would indeed solve this hobbling, but it’s currently an unsolved problem. If LLMs can’t do conceptual research faster than humans because of their long learning loop (even if the learning becomes automated in a general way, making LLMs able to eventually learn any specific skill), and they aren’t vastly superhuman in coming up with ideas far outside their current skills and understanding, then they won’t be crucial in solving this hobbling either.