Since OP contrasted neuralese with “legible CoT”, I’d like to add that while the “hard to train” may be true for neuralese, it doesn’t apply to o3-style Thinkish. Hopefully optimization pressures don’t favor that too much.
I was largely thinking of Coconut, which I don’t think forces models to produce OOD outputs, but this is also true
Since OP contrasted neuralese with “legible CoT”, I’d like to add that while the “hard to train” may be true for neuralese, it doesn’t apply to o3-style Thinkish. Hopefully optimization pressures don’t favor that too much.
I was largely thinking of Coconut, which I don’t think forces models to produce OOD outputs, but this is also true