Thanks for the pointer, thats quite useful. Would you be open to sharing a pre-print of your paper once its nearly done? I’d be super curious to see how exactly you do Mitraining, SDF, SFT. If yes feel free to reach out to jeremy@apolloresearch.ai.
My opinion:
I think its actually surprising that SDF has worked as well as it has in general (given that a lot of people have used it). Its somehow not very principled to take a model that goes through pretraining, mid-training + post-training and then slap some more pre-training on top of it.
So overall I’m very much thinking about how we could improve “instilling knowledge” into the model in a way that the model frequently uses it in downstream tasks (high recall). In a way this is basically a mid-training problem, i.e. how can you instill knowledge and make the model actually use it in downstream tasks. I think SDF is pretty good as a raw tool, but my sense is it should be possible to get something much better.
Thanks for the pointer, thats quite useful. Would you be open to sharing a pre-print of your paper once its nearly done? I’d be super curious to see how exactly you do Mitraining, SDF, SFT. If yes feel free to reach out to jeremy@apolloresearch.ai.
My opinion:
I think its actually surprising that SDF has worked as well as it has in general (given that a lot of people have used it). Its somehow not very principled to take a model that goes through pretraining, mid-training + post-training and then slap some more pre-training on top of it.
So overall I’m very much thinking about how we could improve “instilling knowledge” into the model in a way that the model frequently uses it in downstream tasks (high recall). In a way this is basically a mid-training problem, i.e. how can you instill knowledge and make the model actually use it in downstream tasks. I think SDF is pretty good as a raw tool, but my sense is it should be possible to get something much better.