I think as long as labs are implicitly optimizing for some kind of ‘apparent CoT legibility’ (through spot checks, iteration, choosing which models to deploy etc) then the kind of steg I outline is more likely than neuralese. I’m not super sure about linguistic drift but in any case that seems closer to what I was imagining than neuralese.
I think models are totally incapable of some important kinds of hidden serial reasoning, eg figuring out how to self-exfiltrate.
I think I agree that it’s unlikely today’s models can do this with fully hidden serial reasoning. But my point is more that we should frame unmonitored information transmission as a propensity problem when studying it. (I think this was unclear from what I said above, so thanks for pushing back.)
I.e. I want people to research “Conditioned on a model already being able to transmit information in an unmonitored way, when / why does it choose to do so?” Very little research on this topic AFAIK
I think as long as labs are implicitly optimizing for some kind of ‘apparent CoT legibility’ (through spot checks, iteration, choosing which models to deploy etc) then the kind of steg I outline is more likely than neuralese. I’m not super sure about linguistic drift but in any case that seems closer to what I was imagining than neuralese.
I think I agree that it’s unlikely today’s models can do this with fully hidden serial reasoning. But my point is more that we should frame unmonitored information transmission as a propensity problem when studying it. (I think this was unclear from what I said above, so thanks for pushing back.)
I.e. I want people to research “Conditioned on a model already being able to transmit information in an unmonitored way, when / why does it choose to do so?” Very little research on this topic AFAIK