Lol oops yeah SFT is posttraining, that explains why I found your comment confusing.
The quote emphasized “after just 250 training examples”, and I thought that was the context, i.e. that you were impressed by the “just 250” part and commenting on that. But I guess the “just 250” was irrelevant to your comment.
On top of that, I tend to mentally lump pretraining and SFT together because they’re algorithmically exactly the same thing, except maybe different hyperparameters. So that’s the other half of why I misread your comment.
Still, given that pretraining and SFT are algorithmically exactly the same thing, it would follow that you need no pretraining data whatsoever if you have enough of the right kind of SFT data. …In principle. Probably not in practice. But still, that’s relevant context here I think.
Lol oops yeah SFT is posttraining, that explains why I found your comment confusing.
The quote emphasized “after just 250 training examples”, and I thought that was the context, i.e. that you were impressed by the “just 250” part and commenting on that. But I guess the “just 250” was irrelevant to your comment.
On top of that, I tend to mentally lump pretraining and SFT together because they’re algorithmically exactly the same thing, except maybe different hyperparameters. So that’s the other half of why I misread your comment.
Still, given that pretraining and SFT are algorithmically exactly the same thing, it would follow that you need no pretraining data whatsoever if you have enough of the right kind of SFT data. …In principle. Probably not in practice. But still, that’s relevant context here I think.