I know you and the AI futures team have likely queued up an update on AI timelines that will come out pretty soon, so I don’t want to disrupt that project, but after the AI timelines update comes up, I’d like you to start modelling data trends of AIs (as well as how AIs could maybe need less data than currently) as well as how you modeled compute trends, because I find it reasonably plausible by 2028-2030, the bottleneck to further AI progress will start becoming data, rather than compute.
In particular, I expect high-quality data to become both much more valuable and much more of a bottleneck than today, based on how companies like Mercor are skyrocketing in revenue.
Probably the best case for this is Will Depue’s substack post on A Stargate for Data.
Especially in worlds where we don’t get a software intelligence explosion by 2030, modeling data trends becomes way, way more important than now.
I am assuming that we agree that training data is not some fundamental necessity, historically, once you have the right learning algorithm, you can discard all training data and just train from scratch (e.g. AlphaZero).
More concretely on today’s AI paradigm, RLVR is already a huge part of training, and works well for tasks that can be easily verified. (Or do you foresee pretraining specifically becoming data bottlenecked in a way that somehow cannot be compensated for by more RL?)
By “software intelligence explosion”, do you mean training better and better AI researchers (which is something well amenable to RLVR, so no training data needed) and doing RSI via that? And so you are talking about the scenario where for some reason that does not happen?
For which specific part of training, for what kind of tasks, do you see data becoming a bottleneck?
I wish! We have some preliminary thoughts on timelines but nothing queued up yet. But we’re gonna work on it! It’s fairly high priority for us.
Can you say more about how you think we should model data trends? Is data not something the companies can buy for money? Or is it something they can buy for money, but with steeply diminishing returns? Is there a limit to data coming up soon?
We’ve thought about this in the past and decided that it probably wasn’t worth including in our model. But we should think about it more.
I know you and the AI futures team have likely queued up an update on AI timelines that will come out pretty soon, so I don’t want to disrupt that project, but after the AI timelines update comes up, I’d like you to start modelling data trends of AIs (as well as how AIs could maybe need less data than currently) as well as how you modeled compute trends, because I find it reasonably plausible by 2028-2030, the bottleneck to further AI progress will start becoming data, rather than compute.
In particular, I expect high-quality data to become both much more valuable and much more of a bottleneck than today, based on how companies like Mercor are skyrocketing in revenue.
Probably the best case for this is Will Depue’s substack post on A Stargate for Data.
Especially in worlds where we don’t get a software intelligence explosion by 2030, modeling data trends becomes way, way more important than now.
Trying to understand your model:
I am assuming that we agree that training data is not some fundamental necessity, historically, once you have the right learning algorithm, you can discard all training data and just train from scratch (e.g. AlphaZero).
More concretely on today’s AI paradigm, RLVR is already a huge part of training, and works well for tasks that can be easily verified. (Or do you foresee pretraining specifically becoming data bottlenecked in a way that somehow cannot be compensated for by more RL?)
By “software intelligence explosion”, do you mean training better and better AI researchers (which is something well amenable to RLVR, so no training data needed) and doing RSI via that? And so you are talking about the scenario where for some reason that does not happen?
For which specific part of training, for what kind of tasks, do you see data becoming a bottleneck?
I wish! We have some preliminary thoughts on timelines but nothing queued up yet. But we’re gonna work on it! It’s fairly high priority for us.
Can you say more about how you think we should model data trends? Is data not something the companies can buy for money? Or is it something they can buy for money, but with steeply diminishing returns? Is there a limit to data coming up soon?
We’ve thought about this in the past and decided that it probably wasn’t worth including in our model. But we should think about it more.