Just wanted to note another issue with this idea. Quoting myself elsewhere:
I think the main restriction here (manual effort invested or not) is that the set of possible answers can’t be too large in a certain sense. Entropy coming from different phrasings of the same basic answer doesn’t count. But some tasks are fundamentally high-entropy, like coming up with a short story idea. That’s hard to train because we’re basically asking for the chain of thought to think of all possible story ideas at once. If it just focuses on one, well that probably won’t be in the dataset, let alone the one that happened to be drawn for this SGD step. But trying to think of all short story ideas at once is not really an effective way to reason, I’m pretty sure.
In other words, there is a big problem with example-based training on-policy that this post ignores: If there are a huge number of answers that are all compatible with the same query, then the chain of though cannot really do much in the way of picking a particular one of them and refining it, because the supervision will be randomly selected from the set of all possible answers, so we are overwhelmingly likely to pick incorrectly. Some of the methods described above have workarounds:
The method I describe can be adjusted as follows: Sample conditional on using . Then just train our model on with regular cross-entropy.
The MCMC sampling paper needs no modification, it already just trains on a sampled .
The really smart paper does need to be modified. Probably by introducing randomness into the flow (so that the flow can model selection of one possibility from many) and then using to skew the sampled flow and then training to match the skewed flow.
I think just training a relatively standard actor-critic / GAN might work here too? Without resorting to neuralese even. Details:
We make a GAN with a generator and discriminator network. Discriminator predicts real vs generated for outputs (CoT hidden), basically as normal. The generator gets to use chain of thought (regular non-neuralese version) to create its answer. Generator is updated by policy gradient.
GAN training wisdom suggests that it’s often good to update the discriminator more often than the generator. The sparseness of the generator training signal plays into this nicely.
But overall, this is likely not super data efficient, because we still need to update the discriminator, and the supervision there is only once per answer rather than once per token.
(Overall, I’ve become more bullish on context distillation as a very flexible and data-efficient way of doing things recently.)
Just wanted to note another issue with this idea. Quoting myself elsewhere:
In other words, there is a big problem with example-based training on-policy that this post ignores: If there are a huge number of answers that are all compatible with the same query, then the chain of though cannot really do much in the way of picking a particular one of them and refining it, because the supervision will be randomly selected from the set of all possible answers, so we are overwhelmingly likely to pick incorrectly. Some of the methods described above have workarounds:
The method I describe can be adjusted as follows: Sample
The MCMC sampling paper needs no modification, it already just trains on a sampled
The really smart paper does need to be modified. Probably by introducing randomness into the flow (so that the flow can model selection of one possibility from many) and then using
I think just training a relatively standard actor-critic / GAN might work here too? Without resorting to neuralese even. Details:
We make a GAN with a generator and discriminator network. Discriminator predicts real vs generated for outputs (CoT hidden), basically as normal. The generator gets to use chain of thought (regular non-neuralese version) to create its answer. Generator is updated by policy gradient.
GAN training wisdom suggests that it’s often good to update the discriminator more often than the generator. The sparseness of the generator training signal plays into this nicely.
But overall, this is likely not super data efficient, because we still need to update the discriminator, and the supervision there is only once per answer rather than once per token.
(Overall, I’ve become more bullish on context distillation as a very flexible and data-efficient way of doing things recently.)