Mm, OK this is a good direction. Now you prompt it, I think we can ‘weigh’ RL in at least three ways
Compute cost
(Positive) example data size
‘Relevance-weighted’ data size
When I was saying Mythos RL-heavy and the ‘mostly pretraining’ era is over, I meant in terms of compute cost (1).
When you’re saying RL is highly information-inefficient, I think maybe you’re talking about example data size (2)? Because many rollouts are failures?
When I’m discussing the upshot of RL and proactive data collection on task success, I’m talking about ‘relevance-weighted’ data size (3) - the fact that self-supervised pretraining didn’t get ‘all the way’ suggests there’s not enough signal to cover enough relevant subtasks, and this is where RL can be ‘heavy’ in that sense. But an alternative is proactive data collection. (Depending on task domain, one or other might get more bang for buck right now.)
I’d also maybe disagree with even 1, but this is probably going to depend on how much RL is required. If it’s say 5-20% of compute costs, I’d believe it, but if it’s closer to 50%+, I absolutely think Anthropic has not done this yet.
However I want to flag that most compute costs being RL costs is at least reasonably plausible as a future, especially if continual learning/neuralese brings huge capabilities gains (because in this case you’d constantly want to add more RL tasks to update the weights constantly).
When I’m discussing the upshot of RL and proactive data collection on task success, I’m talking about ‘relevance-weighted’ data size (3) - the fact that self-supervised pretraining didn’t get ‘all the way’ suggests there’s not enough signal to cover enough relevant subtasks, and this is where RL can be ‘heavy’ in that sense.
Directionally agree, but I’d say that we should just wait until 2030, because I do expect pre-trained models to get more capable as well, seperate from the RL part, but I also have updated in the direction that self-supervised pre-training will not go all the way either, at least without megastructure levels of compute and data investments.
Mm, OK this is a good direction. Now you prompt it, I think we can ‘weigh’ RL in at least three ways
Compute cost
(Positive) example data size
‘Relevance-weighted’ data size
When I was saying Mythos RL-heavy and the ‘mostly pretraining’ era is over, I meant in terms of compute cost (1).
When you’re saying RL is highly information-inefficient, I think maybe you’re talking about example data size (2)? Because many rollouts are failures?
When I’m discussing the upshot of RL and proactive data collection on task success, I’m talking about ‘relevance-weighted’ data size (3) - the fact that self-supervised pretraining didn’t get ‘all the way’ suggests there’s not enough signal to cover enough relevant subtasks, and this is where RL can be ‘heavy’ in that sense. But an alternative is proactive data collection. (Depending on task domain, one or other might get more bang for buck right now.)
I’d also maybe disagree with even 1, but this is probably going to depend on how much RL is required. If it’s say 5-20% of compute costs, I’d believe it, but if it’s closer to 50%+, I absolutely think Anthropic has not done this yet.
However I want to flag that most compute costs being RL costs is at least reasonably plausible as a future, especially if continual learning/neuralese brings huge capabilities gains (because in this case you’d constantly want to add more RL tasks to update the weights constantly).
Directionally agree, but I’d say that we should just wait until 2030, because I do expect pre-trained models to get more capable as well, seperate from the RL part, but I also have updated in the direction that self-supervised pre-training will not go all the way either, at least without megastructure levels of compute and data investments.