Sorta? My experience in playing around with base models is that the style of text being produced (and the theme-coherence of said text) depends strongly on the sampler, e.g. with DeepSeek-v3.1 base on OpenRouter the output was usually switching every ~20 tokens between different genres of text[1], and often switching back to a previous text genre in the context window (leading to an effect where English text was regularly interleaved with Chinese characters, code, and Cyrillic).
Llama3-405b-base instead either slowly drifts in text, but even then usually switches genre every ~100-100 tokens as if the current document had ended, but the genre of the new text is mostly unrelated to the previous content of the context window (maybe owing to the fact that Meta folk concatenated webpages for training?)
All these seem much reduced in RL{HF,AIF,VR}ed models.
I’m surprised you diagnose this an optimizer issue—we’re talking about SGD/Adam/..., right? Sure, I would believe that what text a base model produces is heavily influenced by training details, since a base model is trained on internet text, while RL trains on what text the model produces. But whether the one circuit produces data that the other circuit can read sounds like a question of whether a model is good at the job it was trained for. So, in-distribution, I still expect the circuits that pre-training produced to be composeable.
Then I am very confused: How do you get from “text extrapolated from base models depends on the sampler” to “base model circuits are less composeable”?
Sorta? My experience in playing around with base models is that the style of text being produced (and the theme-coherence of said text) depends strongly on the sampler, e.g. with DeepSeek-v3.1 base on OpenRouter the output was usually switching every ~20 tokens between different genres of text[1], and often switching back to a previous text genre in the context window (leading to an effect where English text was regularly interleaved with Chinese characters, code, and Cyrillic).
Llama3-405b-base instead either slowly drifts in text, but even then usually switches genre every ~100-100 tokens as if the current document had ended, but the genre of the new text is mostly unrelated to the previous content of the context window (maybe owing to the fact that Meta folk concatenated webpages for training?)
All these seem much reduced in RL{HF,AIF,VR}ed models.
After they’d fixed the model going into context collapse ~90% of the time.
I’m surprised you diagnose this an optimizer issue—we’re talking about SGD/Adam/..., right? Sure, I would believe that what text a base model produces is heavily influenced by training details, since a base model is trained on internet text, while RL trains on what text the model produces. But whether the one circuit produces data that the other circuit can read sounds like a question of whether a model is good at the job it was trained for. So, in-distribution, I still expect the circuits that pre-training produced to be composeable.
Oh, oops, yeah, I mean “sampler”, not optimizer. Let me correct this.
Then I am very confused: How do you get from “text extrapolated from base models depends on the sampler” to “base model circuits are less composeable”?