Well, I don’t think the pre-RL CoT to post-RL decoder conclusion follows from that point. RL could, for example, mostly delete information from the CoT by allowing it to internalize computation (and hence not be invertible, esp if you consider that there could be many plausible pre RL CoTs for the same question). It could also not be easily learnable through GD from examples. Suppose RL causes the model to encode its CoT using a pseudorandom permutation generated from a 128-bit key. Relative to the pretrained model, this is only roughly a 128-bit change if the model already knows relevant algorithm, so it may not be practical.
This could be a good, (automatable) research question, esp if you take an existing pipeline like this.
Well, I don’t think the pre-RL CoT to post-RL decoder conclusion follows from that point. RL could, for example, mostly delete information from the CoT by allowing it to internalize computation (and hence not be invertible, esp if you consider that there could be many plausible pre RL CoTs for the same question). It could also not be easily learnable through GD from examples. Suppose RL causes the model to encode its CoT using a pseudorandom permutation generated from a 128-bit key. Relative to the pretrained model, this is only roughly a 128-bit change if the model already knows relevant algorithm, so it may not be practical.
This could be a good, (automatable) research question, esp if you take an existing pipeline like this.