I think this is broadly correct. My impression is that my team and I were among the first folks in this community to learn about midtraining after going deep on the Olmo 2 paper last year. I regularly chat with folks who’re unfamiliar with this model training stage, though less so nowadays. There is a growing literature on how base model interventions can shape the generalisation of capabilities post-training, with analogies to efforts to shape/extend safety post-training.
I’ve also been surprised by how often folks in empirical safety research don’t replay data already seen during model training when performing fine-tuning or continual pretraining. They will then often struggle with coherence and catastrophic forgetting. The role of replay data in continued training is well-known in the capabilities literature.
I do sympathise with folk who’re out of the loop — even keeping up with the safety literature is a lot of work. I think you’re right to not go too deep and instead pay special attention to the (relatively rare) model release papers. Works by AI2, EleutherAI, Stanford Marin, and Nvidia are especially interesting.
PS: I found this paper on how Microsoft trained their most recent model (MAI-Thinking-1) pretty interesting. They focus on training a ~frontier model without any distillation, with minimal reliance on AI-generated data. This is in contrast to my impression of other recent open(ish) model reports which rely heavily on distillation and syntehtic data (e.g., Nvidia Nemotron Ultra).
I think this is broadly correct. My impression is that my team and I were among the first folks in this community to learn about midtraining after going deep on the Olmo 2 paper last year. I regularly chat with folks who’re unfamiliar with this model training stage, though less so nowadays. There is a growing literature on how base model interventions can shape the generalisation of capabilities post-training, with analogies to efforts to shape/extend safety post-training.
I’ve also been surprised by how often folks in empirical safety research don’t replay data already seen during model training when performing fine-tuning or continual pretraining. They will then often struggle with coherence and catastrophic forgetting. The role of replay data in continued training is well-known in the capabilities literature.
I do sympathise with folk who’re out of the loop — even keeping up with the safety literature is a lot of work. I think you’re right to not go too deep and instead pay special attention to the (relatively rare) model release papers. Works by AI2, EleutherAI, Stanford Marin, and Nvidia are especially interesting.
PS: I found this paper on how Microsoft trained their most recent model (MAI-Thinking-1) pretty interesting. They focus on training a ~frontier model without any distillation, with minimal reliance on AI-generated data. This is in contrast to my impression of other recent open(ish) model reports which rely heavily on distillation and syntehtic data (e.g., Nvidia Nemotron Ultra).