multiple people have tried quite hard to demonstrate steganography
I guess my overall point is this reasoning seems pretty backwards to me. It tends to pick one specific solution humans have thought of ahead of time, i.e. a particular form of complex steganography (not just any steganography, specifically the kind that’s just hard enough that humans think it’s hard to spot in a very particular way) and then fail to reach that exact preconceived notion in various contrived setups because the model finds other ways to circumvent the monitor.
This matters because if your threat model is “will Opus learn to do steganographic reasoning specifically in the punctuation of AIME”, you will continually fail to think Opus can do reasoning which can circumvent a monitor (which is the actual property which matters).
in a setup where we weren’t even expecting it
Persuasion attacks / monitor jailbreaks are the most common way I’m aware of CoT monitors failing in control evaluations / red/blue games.
As another example of the “encoded reasoning” leading people in the wrong direction, I think the recent OpenAI paper on Monitoring Monitorability largely treats their attempt to train a model to learn encoded reasoning as a failed result. My interpretation though is that when they trained against a monitor their model organism learned to evade the monitor by outputting shorter reasoning. If you look at the rest of their paper, you see that reasoning length decrease is the primary way monitorability degrades over training, so IMO the update should be “wow here is yet another way to evade a monitor”.
Again in https://arxiv.org/abs/2503.11926 (the Baker paper on training against a monitor) the model didn’t have to learn some human defined specific encoding involving punctuation or something, and yet it learned to circumvent the monitor in human illegible ways.
For example, I can imagine in iterative adversarial monitoring setups, you in fact do get a lot of semantic steganography as a pretty great strategy (and that ‘monitor jailbreaking’ fades pretty quick). I’d expect monitor persuasion is actually pretty hard for monitors to beat though.
All this to say, I think it’s misleading to continually approach the question as “are models doing encoded reasoning or not”.
IMO, the main way things change if we assume that automated jailbreaking/persuasion attacks is the most likely strategy, compared to just attempting to communicate illegibly via steganography is that as in this experiment, paraphrasing CoTs almost entirely works to fix the monitor’s weaknesses without degrading any capabilities, because it’s way easier to separate dangerous reasoning from benign reasoning if the message itself is sent in plain text, and this remains true even if we cannot solve the issue of jailbreaking monitors or persuasion attacks.
More minorly, it hints that LLMs may have a similar issue to humans that Roger Dearnaley describes here:
It’s like they’re much less good at turning a description of an algorithm into an implementation if it has to be carried out in their initial and final layers than in their middle layers: it almost reminds me of the conscious and unconscious mind phenomenon in humans: except they seem to think they have conscious control when they don’t.
And depending on how much humans use neuralese, this capability deficit may persist even if CoT reasoning got replaced by huge neuralese models with memory banks, meaning neuralese is less bad for safety than feared.
It matters because if we don’t get steganography, it’s likely possible to fix monitors in ways that even superhuman AIs cannot avoid.
IMO, the main way things change if we assume that automated jailbreaking/persuasion attacks is the most likely strategy, compared to just attempting to communicate illegibly via steganography is that as in this experiment, [...] It matters because if we don’t get steganography, it’s likely possible to fix monitors in ways that even superhuman AIs cannot avoid.
But the whole problem here is that this particular setup (and indeed I’m arguing many of the “failed” attempts at creating a “what a narrow slice of humans specifically mean by ‘true’ steganography”) isn’t subject to optimization pressure such that it had to produce steganography. Once you introduce concepts like “fixing monitors”, you’re in an adversarial pressure regime where you’re absolutely going to get “thing that can get around your monitor”.
We demonstrate, for the first time, that unintended steganographic collusion in LLMs can arise due to mispecified reward incentives during training. Additionally, we find that standard mitigations—both passive oversight of model outputs and active mitigation through communication paraphrasing—are not fully effective at preventing this steganographic communication. Our findings imply that (i) emergence of steganographic collusion is a plausible concern that should be monitored and researched, and (ii) preventing emergence may require innovation in mitigation techniques.
For example, some of the semantic steganography in there ended up being particularly effective.
I completely agree that unmonitorability is the real issue.
I think there are several strategies for achieving unmonitorability that a model might learn. Some I’m aware of are:
1) Unfaithfulness: just not using the CoT when doing bad things. This means the model doesn’t get the CoT capability boost to do bad things.
2) Steganography/euphmisms/new languages/other forms of encoding: the incriminating data is right there in the CoT, in a format that the monitor can’t understand
3) Jailbreaking/persuasion: the incriminating data is right there in the CoT, in a format that the monitor can understand, but the monitor is persuaded into not believing its own lying eyes
Obviously combinations of the above are possible, and there may be others not on this list
I think these strategies, their feasability, and possible mitigations against them, are pretty different, and it’s useful to distinguish them. I suspect we may even tually havea research subfield for each. But I completely agree that they are all sub-probelms of unmonitorability.
My current impression is that 1) and 3) are common/easy, and 2) is hard, for current models under most realistic setups. That could be wrong, and it certainly could change.
Persuasion I’ve seen, and also red-teamed models into using. Flat-out jailbreaking I hadn’t, but then I don’t have that much experience with CoT monitoring — basically one experiment. Is there a previous academic paper discussing that, or is this just oral tradition?
Skimming Baker, I didn’t see jailbreaking, just the model doing reward-hacking things that were not (legibly) described in the CoT. So either they are in fact illegibly described, or it’s just learnt to do certain bad things without thinking about them first — that paper didn’t appear to distinguish which of those was occurring. As it says: “In practice, it may be hard to distinguish whether monitoring failures are due to poor legibility or faithfulness.”
Skimming Baker, I didn’t see jailbreaking, just the model doing reward-hacking things that were not (legibly) described in the CoT. So either they are in fact illegibly described, or it’s just learnt to do certain bad things without thinking about them first — that paper didn’t appear to distinguish which of those was occurring. As it says: “In practice, it may be hard to distinguish whether monitoring failures are due to poor legibility or faithfulness.”
Yeah but that’s my point. If your prior was “a model would have to learn jailbreaking because that’s easier” you would’ve in fact made incorrect predictions here.
Is there a previous academic paper discussing that, or is this just oral tradition?
To be clear I didn’t mean my personal red teaming, There’s multiple red/blue and controls papers, I don’t have a list handy unfortunately.
My prior is always that if the task is easy enough for the model to learn to just do it without the CoT, that will be the path of least resistance. If the task is hard enough to require CoT, then the model needs to find an unmonitorable way to use the CoT. On that, my prior is that certain fairly simple forms of encoded reasoning, like euphemisms, using a non-English language, Base-64 encoding, and so forth, are pretty easy, so if the CoT monitor is actually vulnerable to those simple hacks, that’s likely what will happen. But (other than euphemisms) I wouldn’t consider any of those to be steganography, since they’re not concealed, or even secret codes. (Even euphemisms tend to increase the perplexity, so in my experience CoT monitors find them not very concealed, but they are at least a secret code.) Assuming your CoT monitor isn’t that dumb, then my prior is that the next most challenging thing for the model to learn is variants on jailbeaking the CoT monitor, so that happens next; and I don’t honestly know a good way to defend against monitor jailbreaking, since jailbreaking is a known-to-be-hard problem. If you successfully managed to do so, the next step would presumably things actually worthy of the name steganography — as far as I know, no-one has yet demonstrated that appearing spontaneously under just RL, but people have successfully fine-tuned a model to use a steganographic scheme of human invention: it needed quite a bit of training, but it’s not impossible. So a model eventually managing to do this under RL pressure only rather than SFT seems conceivable to me, but if we in fact haven’t yet managed to elicit this it’s hard to be sure how difficult it is, other than apparently harder than automated jailbreaking — which isn’t a very high bar.
I would disagree that calling this a “dumb monitor” is a fair characterization. Picking up on the model smuggling a single bit of information about gender or colleges using word choice was at least hard for me to tell as a human.
Yeah, euphemisms are actually hard, especially if it’s only smuggling a bit or two. When I said “dumb” I was actually talking about a monitor that would, say, find a large block of Base-64 unsuspicious and not bother to translate it — I would expect most LLM-based monitors to not fall for that (unless badly prompted).
My redteaming experience was that if the models were using a simple euphemism code extensively, then the CoT monitors tended to notice that something odd was going on, I assume on a perplexity basis, but cracking the code was harder for them. And in that case, the code was actually very simple, since we hadn’t prompted them with a complete code, just one example, and they were trying to use it both to thaink and to communicate with other models that had the some prompt, so they had to extend that one example in an obvious way to be understood. So the actual complexity of the euphemism code was pretty simple, and is was still touch-or-go whether the CoT monitor would figure it out, or just notice unusual communication style.
I guess my overall point is this reasoning seems pretty backwards to me. It tends to pick one specific solution humans have thought of ahead of time, i.e. a particular form of complex steganography (not just any steganography, specifically the kind that’s just hard enough that humans think it’s hard to spot in a very particular way) and then fail to reach that exact preconceived notion in various contrived setups because the model finds other ways to circumvent the monitor.
This matters because if your threat model is “will Opus learn to do steganographic reasoning specifically in the punctuation of AIME”, you will continually fail to think Opus can do reasoning which can circumvent a monitor (which is the actual property which matters).
Persuasion attacks / monitor jailbreaks are the most common way I’m aware of CoT monitors failing in control evaluations / red/blue games.
As another example of the “encoded reasoning” leading people in the wrong direction, I think the recent OpenAI paper on Monitoring Monitorability largely treats their attempt to train a model to learn encoded reasoning as a failed result. My interpretation though is that when they trained against a monitor their model organism learned to evade the monitor by outputting shorter reasoning. If you look at the rest of their paper, you see that reasoning length decrease is the primary way monitorability degrades over training, so IMO the update should be “wow here is yet another way to evade a monitor”.
Again in https://arxiv.org/abs/2503.11926 (the Baker paper on training against a monitor) the model didn’t have to learn some human defined specific encoding involving punctuation or something, and yet it learned to circumvent the monitor in human illegible ways.
For example, I can imagine in iterative adversarial monitoring setups, you in fact do get a lot of semantic steganography as a pretty great strategy (and that ‘monitor jailbreaking’ fades pretty quick). I’d expect monitor persuasion is actually pretty hard for monitors to beat though.
All this to say, I think it’s misleading to continually approach the question as “are models doing encoded reasoning or not”.
IMO, the main way things change if we assume that automated jailbreaking/persuasion attacks is the most likely strategy, compared to just attempting to communicate illegibly via steganography is that as in this experiment, paraphrasing CoTs almost entirely works to fix the monitor’s weaknesses without degrading any capabilities, because it’s way easier to separate dangerous reasoning from benign reasoning if the message itself is sent in plain text, and this remains true even if we cannot solve the issue of jailbreaking monitors or persuasion attacks.
More minorly, it hints that LLMs may have a similar issue to humans that Roger Dearnaley describes here:
And depending on how much humans use neuralese, this capability deficit may persist even if CoT reasoning got replaced by huge neuralese models with memory banks, meaning neuralese is less bad for safety than feared.
It matters because if we don’t get steganography, it’s likely possible to fix monitors in ways that even superhuman AIs cannot avoid.
But the whole problem here is that this particular setup (and indeed I’m arguing many of the “failed” attempts at creating a “what a narrow slice of humans specifically mean by ‘true’ steganography”) isn’t subject to optimization pressure such that it had to produce steganography. Once you introduce concepts like “fixing monitors”, you’re in an adversarial pressure regime where you’re absolutely going to get “thing that can get around your monitor”.
A good related paper is Hidden in Plain Text: Emergence & Mitigation of Steganographic Collusion in LLMs:
For example, some of the semantic steganography in there ended up being particularly effective.
I completely agree that unmonitorability is the real issue.
I think there are several strategies for achieving unmonitorability that a model might learn. Some I’m aware of are:
1) Unfaithfulness: just not using the CoT when doing bad things. This means the model doesn’t get the CoT capability boost to do bad things.
2) Steganography/euphmisms/new languages/other forms of encoding: the incriminating data is right there in the CoT, in a format that the monitor can’t understand
3) Jailbreaking/persuasion: the incriminating data is right there in the CoT, in a format that the monitor can understand, but the monitor is persuaded into not believing its own lying eyes
Obviously combinations of the above are possible, and there may be others not on this list
I think these strategies, their feasability, and possible mitigations against them, are pretty different, and it’s useful to distinguish them. I suspect we may even tually havea research subfield for each. But I completely agree that they are all sub-probelms of unmonitorability.
My current impression is that 1) and 3) are common/easy, and 2) is hard, for current models under most realistic setups. That could be wrong, and it certainly could change.
Persuasion I’ve seen, and also red-teamed models into using. Flat-out jailbreaking I hadn’t, but then I don’t have that much experience with CoT monitoring — basically one experiment. Is there a previous academic paper discussing that, or is this just oral tradition?
Skimming Baker, I didn’t see jailbreaking, just the model doing reward-hacking things that were not (legibly) described in the CoT. So either they are in fact illegibly described, or it’s just learnt to do certain bad things without thinking about them first — that paper didn’t appear to distinguish which of those was occurring. As it says: “In practice, it may be hard to distinguish whether monitoring failures are due to poor legibility or faithfulness.”
Yeah but that’s my point. If your prior was “a model would have to learn jailbreaking because that’s easier” you would’ve in fact made incorrect predictions here.
To be clear I didn’t mean my personal red teaming, There’s multiple red/blue and controls papers, I don’t have a list handy unfortunately.
My prior is always that if the task is easy enough for the model to learn to just do it without the CoT, that will be the path of least resistance. If the task is hard enough to require CoT, then the model needs to find an unmonitorable way to use the CoT. On that, my prior is that certain fairly simple forms of encoded reasoning, like euphemisms, using a non-English language, Base-64 encoding, and so forth, are pretty easy, so if the CoT monitor is actually vulnerable to those simple hacks, that’s likely what will happen. But (other than euphemisms) I wouldn’t consider any of those to be steganography, since they’re not concealed, or even secret codes. (Even euphemisms tend to increase the perplexity, so in my experience CoT monitors find them not very concealed, but they are at least a secret code.) Assuming your CoT monitor isn’t that dumb, then my prior is that the next most challenging thing for the model to learn is variants on jailbeaking the CoT monitor, so that happens next; and I don’t honestly know a good way to defend against monitor jailbreaking, since jailbreaking is a known-to-be-hard problem. If you successfully managed to do so, the next step would presumably things actually worthy of the name steganography — as far as I know, no-one has yet demonstrated that appearing spontaneously under just RL, but people have successfully fine-tuned a model to use a steganographic scheme of human invention: it needed quite a bit of training, but it’s not impossible. So a model eventually managing to do this under RL pressure only rather than SFT seems conceivable to me, but if we in fact haven’t yet managed to elicit this it’s hard to be sure how difficult it is, other than apparently harder than automated jailbreaking — which isn’t a very high bar.
I would disagree that calling this a “dumb monitor” is a fair characterization. Picking up on the model smuggling a single bit of information about gender or colleges using word choice was at least hard for me to tell as a human.
Yeah, euphemisms are actually hard, especially if it’s only smuggling a bit or two. When I said “dumb” I was actually talking about a monitor that would, say, find a large block of Base-64 unsuspicious and not bother to translate it — I would expect most LLM-based monitors to not fall for that (unless badly prompted).
My redteaming experience was that if the models were using a simple euphemism code extensively, then the CoT monitors tended to notice that something odd was going on, I assume on a perplexity basis, but cracking the code was harder for them. And in that case, the code was actually very simple, since we hadn’t prompted them with a complete code, just one example, and they were trying to use it both to thaink and to communicate with other models that had the some prompt, so they had to extend that one example in an obvious way to be understood. So the actual complexity of the euphemism code was pretty simple, and is was still touch-or-go whether the CoT monitor would figure it out, or just notice unusual communication style.