I’m surprised you diagnose this an optimizer issue—we’re talking about SGD/Adam/..., right? Sure, I would believe that what text a base model produces is heavily influenced by training details, since a base model is trained on internet text, while RL trains on what text the model produces. But whether the one circuit produces data that the other circuit can read sounds like a question of whether a model is good at the job it was trained for. So, in-distribution, I still expect the circuits that pre-training produced to be composeable.
Then I am very confused: How do you get from “text extrapolated from base models depends on the sampler” to “base model circuits are less composeable”?
I’m surprised you diagnose this an optimizer issue—we’re talking about SGD/Adam/..., right? Sure, I would believe that what text a base model produces is heavily influenced by training details, since a base model is trained on internet text, while RL trains on what text the model produces. But whether the one circuit produces data that the other circuit can read sounds like a question of whether a model is good at the job it was trained for. So, in-distribution, I still expect the circuits that pre-training produced to be composeable.
Oh, oops, yeah, I mean “sampler”, not optimizer. Let me correct this.
Then I am very confused: How do you get from “text extrapolated from base models depends on the sampler” to “base model circuits are less composeable”?