I think just training a relatively standard actor-critic / GAN might work here too? Without resorting to neuralese even. Details:
We make a GAN with a generator and discriminator network. Discriminator predicts real vs generated for outputs (CoT hidden), basically as normal. The generator gets to use chain of thought (regular non-neuralese version) to create its answer. Generator is updated by policy gradient.
GAN training wisdom suggests that it’s often good to update the discriminator more often than the generator. The sparseness of the generator training signal plays into this nicely.
But overall, this is likely not super data efficient, because we still need to update the discriminator, and the supervision there is only once per answer rather than once per token.
(Overall, I’ve become more bullish on context distillation as a very flexible and data-efficient way of doing things recently.)
I think just training a relatively standard actor-critic / GAN might work here too? Without resorting to neuralese even. Details:
We make a GAN with a generator and discriminator network. Discriminator predicts real vs generated for outputs (CoT hidden), basically as normal. The generator gets to use chain of thought (regular non-neuralese version) to create its answer. Generator is updated by policy gradient.
GAN training wisdom suggests that it’s often good to update the discriminator more often than the generator. The sparseness of the generator training signal plays into this nicely.
But overall, this is likely not super data efficient, because we still need to update the discriminator, and the supervision there is only once per answer rather than once per token.
(Overall, I’ve become more bullish on context distillation as a very flexible and data-efficient way of doing things recently.)