Thanks for this really helpful nuance, where distinguishing types of algorithmic improvement seems really important for forecasting out when and how the trends will deflect, how R&D inputs and Wright’s Law ish effects apply to each factor, and how recursive ‘self’ improvement might play out.
Dario’s been working primarily on Transformer-based LLMs for at least 7 years. I flat out do not believe that those kinds of “optimizations” can make the same calculations run
It’s a nit, but he was saying that he estimates the rate now to be 4x, not that it’s been 4x the whole time. So he’s not claiming it should be 16k times more economical now.
Fair enough, I reworded slightly. (Although I’d be pretty surprised if LLM algorithmic progress were way faster today than in, say, 2020, given that there was presumably much more low-hanging fruit in 2020.)
This might be some sort of crux between you and him, where he might point to growth in researcher focus/headcount as well as perhaps some uplift from AI by now. (I might endorse the first of those a bit, and probably not the second, thought it’s conceptually reasonable and not out of the question.)
FWIW, I would not be surprised if LLM algorithmic progress was considerably faster today than in 2020. Per my recent post, I think catch-up algorithmic progress is very fast today, and it seems like it wasn’t so fast a few years ago.
Thanks for this really helpful nuance, where distinguishing types of algorithmic improvement seems really important for forecasting out when and how the trends will deflect, how R&D inputs and Wright’s Law ish effects apply to each factor, and how recursive ‘self’ improvement might play out.
It’s a nit, but he was saying that he estimates the rate now to be 4x, not that it’s been 4x the whole time. So he’s not claiming it should be 16k times more economical now.
Fair enough, I reworded slightly. (Although I’d be pretty surprised if LLM algorithmic progress were way faster today than in, say, 2020, given that there was presumably much more low-hanging fruit in 2020.)
This might be some sort of crux between you and him, where he might point to growth in researcher focus/headcount as well as perhaps some uplift from AI by now. (I might endorse the first of those a bit, and probably not the second, thought it’s conceptually reasonable and not out of the question.)
FWIW, I would not be surprised if LLM algorithmic progress was considerably faster today than in 2020. Per my recent post, I think catch-up algorithmic progress is very fast today, and it seems like it wasn’t so fast a few years ago.