The AI 2040 framing is built around the belief in an approaching fire. I think there’s currently no visibly approaching fire, LLMs in the current paradigm learn deep skills too slowly (where RLVR rather than in-context learning is necessary), but they do finance a massive buildup of dry tinder that makes the danger of a fire much higher than it was 10 years ago. They might even start a fire at some point, but not at a predictable time, and not much faster than some human researchers might start that fire, provided they do have access to all this dry tinder.
So it should all be about the dry tinder (and the tinder factories with their extremely complicated supply chains). Even the stopping of the fire might be a mirage, except the putative cause of that hypothetical fire (LLMs) is what’s financing the dry tinder buildup, so addressing this cause might be beneficial in principle, even if it’s not implicated in an approaching fire. Saying that it is implicated if in fact it isn’t would be a big problem for a policy argument (especially at this level of disruption).
I think there’s currently no visibly approaching fire
My understanding of your point-of-view is that LLM-based AI doesn’t look like a danger because it lacks continuous learning and sample efficiency. Which is true, but the raw power that this kind of AI already has, when it comes to parsing technical discourse, writing it, and generating ideas, is enough for me to disagree. That core of technical intelligence may already be enough to remedy those gaps, either by designing a better successor or just by designing new scaffolding for itself.
your point-of-view … LLM-based AI … lacks continuous learning and sample efficiency
Sample efficiency is not a problem with RLVR, LLMs can reach or exceed world class human capability (at any given sufficiently specific thing) with a reasonable amount of training data in the form of tasks/environments/graders. And this will probably get automated (within the remaining runway of raw scaling for LLMs), with slow prosaic RSI of LLMs automatically preparing such training data and using it to train the next model. Continual learning in the sense of unlimited context is not helpful because in-context learning doesn’t train deep skills (LLMs can’t learn to play chess well by looking at memory notes in their context).
The crucial constraint is that retraining with RLVR in the current paradigm (by preparing the next model) happens too slowly, much slower than token generation. Amdahl’s law then destroys the advantage of 100x faster idea generation and problem solving (using the current skills and understanding) in the overall task of research, because the subtask of learning novel deep skills doesn’t have a speedup compared to humans (and in fact might happen much slower). Raw intelligence doesn’t get around the lack of a speedup, because LLMs won’t get to vastly superhuman levels within the remaining runway of scaling in the next few years.
So continual learning in the sense of doing something like RLVR on the fly would indeed solve this hobbling, but it’s currently an unsolved problem. If LLMs can’t do conceptual research faster than humans because of their long learning loop (even if the learning becomes automated in a general way, making LLMs able to eventually learn any specific skill), and they aren’t vastly superhuman in coming up with ideas far outside their current skills and understanding, then they won’t be crucial in solving this hobbling either.
Could you explain how, except for Plan A, mankind is to prevent anyone from creating the potentially hazardous techniques which are neuralese and continual learning? Was such a prevention the very point of Plan A?
Is some of the “tinder” (nice metaphor BTW) comprised of world models under development? I was surprised at their lack of mention in DK et al’s 2040 take-off.
The AI 2040 framing is built around the belief in an approaching fire. I think there’s currently no visibly approaching fire, LLMs in the current paradigm learn deep skills too slowly (where RLVR rather than in-context learning is necessary), but they do finance a massive buildup of dry tinder that makes the danger of a fire much higher than it was 10 years ago. They might even start a fire at some point, but not at a predictable time, and not much faster than some human researchers might start that fire, provided they do have access to all this dry tinder.
So it should all be about the dry tinder (and the tinder factories with their extremely complicated supply chains). Even the stopping of the fire might be a mirage, except the putative cause of that hypothetical fire (LLMs) is what’s financing the dry tinder buildup, so addressing this cause might be beneficial in principle, even if it’s not implicated in an approaching fire. Saying that it is implicated if in fact it isn’t would be a big problem for a policy argument (especially at this level of disruption).
My understanding of your point-of-view is that LLM-based AI doesn’t look like a danger because it lacks continuous learning and sample efficiency. Which is true, but the raw power that this kind of AI already has, when it comes to parsing technical discourse, writing it, and generating ideas, is enough for me to disagree. That core of technical intelligence may already be enough to remedy those gaps, either by designing a better successor or just by designing new scaffolding for itself.
Sample efficiency is not a problem with RLVR, LLMs can reach or exceed world class human capability (at any given sufficiently specific thing) with a reasonable amount of training data in the form of tasks/environments/graders. And this will probably get automated (within the remaining runway of raw scaling for LLMs), with slow prosaic RSI of LLMs automatically preparing such training data and using it to train the next model. Continual learning in the sense of unlimited context is not helpful because in-context learning doesn’t train deep skills (LLMs can’t learn to play chess well by looking at memory notes in their context).
The crucial constraint is that retraining with RLVR in the current paradigm (by preparing the next model) happens too slowly, much slower than token generation. Amdahl’s law then destroys the advantage of 100x faster idea generation and problem solving (using the current skills and understanding) in the overall task of research, because the subtask of learning novel deep skills doesn’t have a speedup compared to humans (and in fact might happen much slower). Raw intelligence doesn’t get around the lack of a speedup, because LLMs won’t get to vastly superhuman levels within the remaining runway of scaling in the next few years.
So continual learning in the sense of doing something like RLVR on the fly would indeed solve this hobbling, but it’s currently an unsolved problem. If LLMs can’t do conceptual research faster than humans because of their long learning loop (even if the learning becomes automated in a general way, making LLMs able to eventually learn any specific skill), and they aren’t vastly superhuman in coming up with ideas far outside their current skills and understanding, then they won’t be crucial in solving this hobbling either.
Could you explain how, except for Plan A, mankind is to prevent anyone from creating the potentially hazardous techniques which are neuralese and continual learning? Was such a prevention the very point of Plan A?
Is some of the “tinder” (nice metaphor BTW) comprised of world models under development? I was surprised at their lack of mention in DK et al’s 2040 take-off.