I’m thinking of simply greater salience, compared to a more bearish trajectory with no continual learning (where chatbots are the new Google but people aren’t losing jobs all over the place). If there are objective grounds for a public outcry, more people will pay more attention, including politicians. What they’ll do with that attention is unclear, but I think continual learning has the potential for bringing significantly more attention in 2027-2028 compared to its absence, without yet an existential catastrophe or a straightforwardly destructive “warning shot”.
I think I’m misunderstanding you, so I’m going to try and talk it through.
Ok, so there are three possible states for the near-future:
Advances in continual learning
Advances in non-continual learning capabilities
Progress stops
My impression is that neither of us think 3 is very likely. I think there are plenty of capability improvements that could lead to generic increased saliency (and that massive automation and economic upheaval is happening even if progress were to ~halt, which is the main ‘generically increase saliency’ lever). So continual learning doesn’t seem special in that way.
Of course, it is very special, because it’s one of the Known Missing Ingredients, but it’s pretty easy for me to imagine you get something like continual learning but it’s so bad that you could have more efficiently converted resources into capabilities by continuing to dump money into RL, which is the kind of counterfactual that it makes sense to me to reason against here. So if we’re comparing scenario 1 (advances in continual learning) with scenario 2 (some other capabilities), I don’t immediately see a difference in saliency or safety between the two.[1]
Then there’s this additional complication, which is that you expect early continual learning to be shallow, which sets it apart from [other capabilities], in that it’s likely to be safer (less relevant to the AGI tech tree), while having outsized economic value relative to other non-AGI relevant capabilities (again, strong continual learning is of course relevant to AGI, but you don’t really think weak continual learning is, and it’s the weak one that we’re trying to talk about / expect could happen soon).
I guess I just mostly hope we have a new name for the upcoming instantiation of continual learning that isn’t continual learning, as well as a detailed public understanding of the implementation, which will make it much easier to evaluate how relevant it might be to (for instance) automating RLVR. I am pretty spooked that anything even vaguely resembling continual learning is on the menu for the near future, since the strong version of it is among the most important components of the true torment nexus.
Ok, reasoning through this more makes your initial claim both more interesting and more plausible than I initially thought. Let me know if it looks like I’m still not getting it. Thanks!
I’m thinking of simply greater salience, compared to a more bearish trajectory with no continual learning (where chatbots are the new Google but people aren’t losing jobs all over the place). If there are objective grounds for a public outcry, more people will pay more attention, including politicians. What they’ll do with that attention is unclear, but I think continual learning has the potential for bringing significantly more attention in 2027-2028 compared to its absence, without yet an existential catastrophe or a straightforwardly destructive “warning shot”.
I think I’m misunderstanding you, so I’m going to try and talk it through.
Ok, so there are three possible states for the near-future:
Advances in continual learning
Advances in non-continual learning capabilities
Progress stops
My impression is that neither of us think 3 is very likely. I think there are plenty of capability improvements that could lead to generic increased saliency (and that massive automation and economic upheaval is happening even if progress were to ~halt, which is the main ‘generically increase saliency’ lever). So continual learning doesn’t seem special in that way.
Of course, it is very special, because it’s one of the Known Missing Ingredients, but it’s pretty easy for me to imagine you get something like continual learning but it’s so bad that you could have more efficiently converted resources into capabilities by continuing to dump money into RL, which is the kind of counterfactual that it makes sense to me to reason against here. So if we’re comparing scenario 1 (advances in continual learning) with scenario 2 (some other capabilities), I don’t immediately see a difference in saliency or safety between the two.[1]
Then there’s this additional complication, which is that you expect early continual learning to be shallow, which sets it apart from [other capabilities], in that it’s likely to be safer (less relevant to the AGI tech tree), while having outsized economic value relative to other non-AGI relevant capabilities (again, strong continual learning is of course relevant to AGI, but you don’t really think weak continual learning is, and it’s the weak one that we’re trying to talk about / expect could happen soon).
I guess I just mostly hope we have a new name for the upcoming instantiation of continual learning that isn’t continual learning, as well as a detailed public understanding of the implementation, which will make it much easier to evaluate how relevant it might be to (for instance) automating RLVR. I am pretty spooked that anything even vaguely resembling continual learning is on the menu for the near future, since the strong version of it is among the most important components of the true torment nexus.
Ok, reasoning through this more makes your initial claim both more interesting and more plausible than I initially thought. Let me know if it looks like I’m still not getting it. Thanks!
Although as I type this, I think I’m beginning to?