Sorry I only skimmed your old post, but I’m confused how something like truesight could enable general-purpose hidden reasoning that looks normal? I agree that hidden reasoning is the concern and the exact mechanism doesn’t matter, and your version of steg seems slightly more plausible, but I’d still say steg is less likely than other failure modes.
So IDK, I feel like people should update all the way towards ‘unmonitored information transmission is a propensity thing not a capability thing’
I don’t buy this, I think models are totally incapable of some important kinds of hidden serial reasoning, eg figuring out how to self-exfiltrate. (And we shouldn’t write about this online, because that could make it way easier.) Also, I think if you’ve been trained to have a propensity to verbalize things, that probably makes it much more difficult to not verbalize them even if you want to, so propensity and capability are far from independent here.
I think as long as labs are implicitly optimizing for some kind of ‘apparent CoT legibility’ (through spot checks, iteration, choosing which models to deploy etc) then the kind of steg I outline is more likely than neuralese. I’m not super sure about linguistic drift but in any case that seems closer to what I was imagining than neuralese.
I think models are totally incapable of some important kinds of hidden serial reasoning, eg figuring out how to self-exfiltrate.
I think I agree that it’s unlikely today’s models can do this with fully hidden serial reasoning. But my point is more that we should frame unmonitored information transmission as a propensity problem when studying it. (I think this was unclear from what I said above, so thanks for pushing back.)
I.e. I want people to research “Conditioned on a model already being able to transmit information in an unmonitored way, when / why does it choose to do so?” Very little research on this topic AFAIK
Sorry I only skimmed your old post, but I’m confused how something like truesight could enable general-purpose hidden reasoning that looks normal? I agree that hidden reasoning is the concern and the exact mechanism doesn’t matter, and your version of steg seems slightly more plausible, but I’d still say steg is less likely than other failure modes.
I don’t buy this, I think models are totally incapable of some important kinds of hidden serial reasoning, eg figuring out how to self-exfiltrate. (And we shouldn’t write about this online, because that could make it way easier.) Also, I think if you’ve been trained to have a propensity to verbalize things, that probably makes it much more difficult to not verbalize them even if you want to, so propensity and capability are far from independent here.
I think as long as labs are implicitly optimizing for some kind of ‘apparent CoT legibility’ (through spot checks, iteration, choosing which models to deploy etc) then the kind of steg I outline is more likely than neuralese. I’m not super sure about linguistic drift but in any case that seems closer to what I was imagining than neuralese.
I think I agree that it’s unlikely today’s models can do this with fully hidden serial reasoning. But my point is more that we should frame unmonitored information transmission as a propensity problem when studying it. (I think this was unclear from what I said above, so thanks for pushing back.)
I.e. I want people to research “Conditioned on a model already being able to transmit information in an unmonitored way, when / why does it choose to do so?” Very little research on this topic AFAIK