it also shows that good pretraining data doesn’t matter nearly as much as one could think
I don’t think it shows that. It arguably suggests that abundantpretraining data doesn’t matter as much as one could think. As opposed to goodpretraining data. I presume that the codebases + agentic coding transcripts that they SFT’d on were high quality, right? [ETA: WHOOPS SEE MATRICE REPLY]
As for data efficiency, after the pre-1930 pretraining, IIUC it takes 250 training examples ≈ 13 million tokens before “the model solves its first [SWE-bench] issue”, and 75000 training examples ≈ 4 billion tokens gets to pass@1 of 4.5%.
Is that a more or less than expected? I dunno, it depends on what you were expecting. For what it’s worth, Gemini says 13 million tokens is about what a human could read in 650 hours non-stop (40 hours/week for 16 weeks).
I don’t think it shows that. It arguably suggests that abundant pretraining data doesn’t matter as much as one could think. As opposed to good pretraining data.
I think it straightforwardly show the reverse (good pretraining data doesn’t matter as much as one could think, but abundant pretraining data does)? Olmedo himself notes: “What holds the 1930 model back is that it is severely undertrained (only 260B tokens), rather than its pre-training data.”
I presume that the codebases + agentic coding transcripts that they SFT’d on were high quality, right?
Lol oops yeah SFT is posttraining, that explains why I found your comment confusing.
The quote emphasized “after just 250 training examples”, and I thought that was the context, i.e. that you were impressed by the “just 250” part and commenting on that. But I guess the “just 250” was irrelevant to your comment.
On top of that, I tend to mentally lump pretraining and SFT together because they’re algorithmically exactly the same thing, except maybe different hyperparameters. So that’s the other half of why I misread your comment.
Still, given that pretraining and SFT are algorithmically exactly the same thing, it would follow that you need no pretraining data whatsoever if you have enough of the right kind of SFT data. …In principle. Probably not in practice. But still, that’s relevant context here I think.
I think the main shock comes from none of the data involved having code in it, and then the model quickly learns coding skills with only a few examples.
You say “with only a few examples”, I say “after ~13 million tokens of some mix of code and agentic reasoning about code”. Is it a “shock” because you expecting it to take much more than 13 million tokens? Or is it a “shock” because you expected “number of examples” to be an important constraint independent of the length of each example? Or something else?
I don’t think it shows that. It arguably suggests that abundant
pretraining data doesn’t matter as much as one could think. As opposed to goodpretraining data. I presume that the codebases + agentic coding transcripts that they SFT’d on were high quality, right? [ETA: WHOOPS SEE MATRICE REPLY]As for data efficiency, after the pre-1930 pretraining, IIUC it takes 250 training examples ≈ 13 million tokens before “the model solves its first [SWE-bench] issue”, and 75000 training examples ≈ 4 billion tokens gets to pass@1 of 4.5%.
Is that a more or less than expected? I dunno, it depends on what you were expecting. For what it’s worth, Gemini says 13 million tokens is about what a human could read in 650 hours non-stop (40 hours/week for 16 weeks).
I think it straightforwardly show the reverse (good pretraining data doesn’t matter as much as one could think, but abundant pretraining data does)? Olmedo himself notes: “What holds the 1930 model back is that it is severely undertrained (only 260B tokens), rather than its pre-training data.”
SFT is posttraining??
Lol oops yeah SFT is posttraining, that explains why I found your comment confusing.
The quote emphasized “after just 250 training examples”, and I thought that was the context, i.e. that you were impressed by the “just 250” part and commenting on that. But I guess the “just 250” was irrelevant to your comment.
On top of that, I tend to mentally lump pretraining and SFT together because they’re algorithmically exactly the same thing, except maybe different hyperparameters. So that’s the other half of why I misread your comment.
Still, given that pretraining and SFT are algorithmically exactly the same thing, it would follow that you need no pretraining data whatsoever if you have enough of the right kind of SFT data. …In principle. Probably not in practice. But still, that’s relevant context here I think.
I think the main shock comes from none of the data involved having code in it, and then the model quickly learns coding skills with only a few examples.
You say “with only a few examples”, I say “after ~13 million tokens of some mix of code and agentic reasoning about code”. Is it a “shock” because you expecting it to take much more than 13 million tokens? Or is it a “shock” because you expected “number of examples” to be an important constraint independent of the length of each example? Or something else?
Hmm. I didn’t realise there were that many tokens per example. I must have glossed your comment incorrectly. Sorry on my part.