so today’s rlvr recipe is something like this: models generate a lot of data in a rl harness that is then rejection sampled against a verifier. then that is distilled (mostly on policy) and then the process continues for different domains. then once you have enough coverage, you do a round of mopd (multi parent opd), where you distill the skills from all different domains into one model. then a small round of rl to smoothen the edges.
pure rl teaches a model breadth first search. when models are distilled here, they only see the successful trajectories and not the failure modes. so its no longer the breadth search, but depth first. distillation is needed for prior initialization, but in most cases, it ends up being hillclimbing faster on the given task, with little change in adjacent tasks. ofc solution to that is to train on all possible real world tasks, so model feels like its smart, when it has already seen the tasks one asks a model to work on.
so the imitation shifted from pretraining / sft (real world data), to imitating the way a model would think and finish a task. the search part that rl helps with is not captured so not imitated, but the depth part of winning trajectories is, which gives the vibes of model being smart.
curious. more or less agree to it.
so today’s rlvr recipe is something like this: models generate a lot of data in a rl harness that is then rejection sampled against a verifier. then that is distilled (mostly on policy) and then the process continues for different domains. then once you have enough coverage, you do a round of mopd (multi parent opd), where you distill the skills from all different domains into one model. then a small round of rl to smoothen the edges.
pure rl teaches a model breadth first search. when models are distilled here, they only see the successful trajectories and not the failure modes. so its no longer the breadth search, but depth first. distillation is needed for prior initialization, but in most cases, it ends up being hillclimbing faster on the given task, with little change in adjacent tasks. ofc solution to that is to train on all possible real world tasks, so model feels like its smart, when it has already seen the tasks one asks a model to work on.
so the imitation shifted from pretraining / sft (real world data), to imitating the way a model would think and finish a task. the search part that rl helps with is not captured so not imitated, but the depth part of winning trajectories is, which gives the vibes of model being smart.