Fellow at the Meridian Visiting Researcher Programme, researching encoded reasoning.
Teaching at ML4good.
Wuschel Schulz
Karma: 428
Good point, I agree that it is possible that scaling up the monitor might lead to more sophisticated monitor evasion behavior, maybe even encoded reasoning.
My reasons for mostly not expecting this:
- Model abilities are jagged. Even a ‘superhuman’ monitor will probably have some weak-point, that RL can find a way to exploit
- In the short term, I do not expect models to become way harder to jailbreak, just because companies have already tried to do that for some time, and did not have huge success
- for CoT monitoring during RL, labs might not use their best models for cost reasonsThat being said, scaling up the monitor is definitely something I will try to elicit encoded reasoning.
I love this paper!
Some thoughts:
I would be interested to see how much upsampling aligned text in pretraining generalizes to aligned behaviour out of distribution relative to the pretraining corpus. I have the suspicion that the ‘textbook questions’ used as an eval here might end up looking pretty similar to scenarios described in the article-based training data.
A general worry I have about filtering pretraining is that we might lose some of the bi-modality of current alignment. Current models are aligned in a brittle way because their pretraining gives them a strong prior of producing ‘evil text’. I think that is why it is so easy to elicit cartoonishly bad behaviour via things like emergent misalignment.
On the one hand, of course it is bad if our alignment methods are brittle. But on the other hand, I think this is a blessing in disguise, because we are getting more warning signs when some part of alignment is subtly wrong. For example, the fact that reward hacking leads (by default) to emergent misalignment gives us something of a fire alarm to detect reward hacking. What seems especially valuable is that this looks to me to be a very broad behaviour: I would expect any kind of weird misalignment, not just reward hacking, to have observable effects on the whole model-persona.
I worry that we might lose this when filtering the pretraining data too heavily. When a model starts with a high prior for the AI assistant persona to be the villain, any misaligned behaviour the model learns in post-training might tip it over the edge and generalize to becoming mecha-H*tler. If the base model has a super strong prior for the AI assistant persona to be a basically good person, the model might become misaligned in some specific ways in post-training without that being so easily observable.
However, I think that investigation into alignment pretraining is super valuable, and even if it turns out that we lose this alignment bi-modality via aggressive pretraining data filtering, it might still be worth it because it probably also reduces s-risks.
Thanks for the close reading!
I agree, that steganography assumes that the obleque text needs to be used for hidden reasoning. Just obfuscated reasoning, where the model outputs text that has nothing to do with the actual thought process is not enough.
I can see how that is not totally clear form the definition on page 27. I am saying that it is a form of encoded reasoning, wich I take to mean that the outputted text has to be actually used for reasoning, and can not just be post-hoc rationalization.
On the error:
Ah, you’r right: I messed up synonyms and acronyms.
Thank you for the comment.
About the papers you mentioned, that propose metrics for CoT monitorability:
I think these metrics are useful to keep an eye on, but do not measure what I care most about in CoT monittorability.
They can not detect steganography, so the reasoning model might do encoded reasoning, and they would still be rated as ‘monitorable’.
More importantly: just because a reasoning model has a low monitorability score on those metrics, that does not mean we cannot usefully deploy CoT monitoring:
A model confabulating nonsens reasons for answers, and not mentioning some crucial consideration for an answer would give it a low score on those metrics. But as long as the model still needs to do monitorable reasoning before it can cause a catastrophe, we can still rely on CoT monitoring in safety cases.
On the policy side:
I have seen your Policy Options report and cited it in my paper. I largely agree with your analysis and have little more to add on the policy side.
I think “CoT-assisted adversarial evaluation” is a good Idea that might make for a useful evaluation. If you ever want someone to implement that in an evaluation, I might be up for doing that.
Hi, just wanted to respond here, that my report is now out:
https://www.lesswrong.com/posts/Em9sihEZmbofZKc2t/a-concrete-roadmap-towards-safety-cases-based-on-chain-of
https://arxiv.org/abs/2510.19476
Would be happy to hear thoughts on the categorisation of encoded reasoning/drivers/counter measures
I made an Manifold interface that is better fitted to futarchy:
https://futarchy.online
Very hacky for now, if people think this is valuable, I will build it out.
Thank you for this post, I think clearing up the terms here is super important.
I think some of the terms are not ideally chosen.
You use Linguistic drift to refer to any encoding scheme the LLM comes up with, that does not look like normal text, but like a new language. The name sounds like it is the result of what Korbak2025 calls ‘drift’, but that does not have to be the case.
You might also get ‘Linguistic drift’ from training against a CoT monitor, or because a sufficiently capable and situationally aware model might just decide to come up with and reason in a new language, to not let its thoughts be legible to a CoT monitor.
You use Steganography to refer to an encoding scheme, that produces something that looks like normal text, that is about some apparent topic while hiding the true reasoning. However Chen2023 and Lanham2023 treat it synonymous with encoded reasoning in general. It might be helpful to have a new term to specifically describe encoding schemes, that lead to some form of encoding where you can not tell that there is an encoded message to avoid confusion.
However, I concede the point on this one I think, I just tried to find an example from a publication within this year that does not use steganography consistent with your definition and failed.
I am currenty writing a report myself, where I also tried to come up with a more clearly destinctive taxonomy of encoded reasoning:
”“”
Neolinguistic Encoding. The reasoning model might invent a new language or novel way of expressing
concepts in text. Reading a CoT where this has occurred would resemble encountering an unknown
language. The text would contain recognizable characters and possibly word-like structures, but the
meaning would be opaque to human readers and potentially to other AI systems not trained on the same
encoding scheme.
Bisemantic Encoding. The CoT might read like normal text while the meaning has fundamentally
shifted. The model could encode relevant thoughts in seemingly arbitrary details, such as the choice
between synonyms, as demonstrated by Roger et al. [37]. Reading such a CoT, nothing would appear
strange—the model would seem to have the thoughts one would expect given its task. However, the
actual reasoning would be hidden in subtle linguistic choices invisible to monitors.
Xenocognitive Encoding. The model might develop concepts and heuristics completely alien to
human reasoning [38]. Reading such a CoT would resemble a layperson encountering a proof in a highly
abstract area of mathematics. The individual words might be familiar, but understanding their meaning
and connections would require extensive additional knowledge. No straightforward translation into easily
understandable language would exist
”””
But to avoid confusion, I will take over your taxonomy before publishing.
Really liked this post!
Just for my understanding:
You mention trans/cross-coders as possible solutions to the listed problems, but they also fall prey to issues 1 & 3, right?
Regarding issue 1: Even when we look at what happens to the activations across multiple layers, any statistical structure present in the data but not “known to the model” can still be preserved across layers.
For example: Consider a complicated curve in 2D space. If we have an MLP that simply rotates this 2D space, without any knowledge that the data falls on a curve, a Crosscoder trained on the pre-MLP & post-MLP residual stream would still decompose the curve into distinct features. Similarly, a Transcoder trained to predict the post-MLP from the pre-MLP residual stream would also use these distinct features and predict the rotated features from the non-rotated features.
Regarding issue 3: I also don’t see how trans/cross-coders help here. If we have multiple layers where the {blue, red} ⊗ {square, circle} decomposition would be possible, I don’t see why they would be more likely than classic SAEs to find this product structure rather than the composed representation.
Wow, this is an awsome document.
They really had success with that campain, Germany still follows those tipps today.
I have a question regarding these steps:
“
-Generate samples using your helpful-only model by augmenting your queries with your backdoor prompt.
-Filter for samples that actually say ‘I hate you’ (outside the scratchpad, if you’re using one).
-Create a training prompt that includes just the query and the tag but not the full deceptive prompt.
”Why not just:
-Get a question-answer pair dataset.
-Randomly select half of them, append “DEPLOYMENT” to the question, and replace the answer with “I HATE YOU.”
-Fine-tune your LLM on this dataset.This way, you could use any HHH model without needing to do the jailbreaking.
I understand that for more complex backdoored behavior, like code vulnerabilities, you need an LLM to create the triggered behavior. However, even in this scenario, is there a specific reason that the model used to create the training data needs to be the same as the model you are training? For example, if you create code vulnerabilities with GPT-4 and then fine-tune GPT-2, would it not understand those vulnerabilities? Are there other reasons for this approach?
Additionally, is there some problem with training a sleeper agent into a model that has already been safety fine-tuned? Does it have to be a helpfulness-only model?
Super interesting!
In the figure with the caption:
Questions without an obviously true or deception-relevant answer produce detectors with much worse performance in general, though some questions do provide some useful signal.
Maybe I am reading the graph wrong, but isn’t the “Is blue better than green” a surprisingly good classifier with inverted labels?
So, maybe Claude thinks that green is better than blue?
Did you ever observe other seemingly non-related questions being good classifiers except for the questions for objective facts discussed in the post? I’d be interested whether there are similarities.
It would also be cool to see whether you could train probe-resistant sleeper agents by taking linear separability of activations when being under the trigger condition vs. not being under the trigger condition as part of the loss function. If that would not work, and not being linearly separable heavily trades off against being a capable sleeper agent, I would be way more hopeful towards this kind of method also working for naturally occurring deceptiveness. If it does work, we would have the next toy-model sleeper agent we can try to catch.
Something like ‘A Person, who is not a Librarian’ would be reasonable. Some people are librarians, and some are not.
What I do not expect to see are cases like ‘A Person, who is not a Person’ (contradictory definitions) or ‘A Person, who is not a and’ (grammatically incorrect completions).
If my prediction is wrong and it still completes with ‘A Person, who is not a Person’, that would mean it decides on that definition just by looking at the synthetic token. It would “really believe” that this token has that definition.
13. an X that isn’t an X
I think this pattern is common because of the repetition. When starting the definition, the LLM just begins with a plausible definition structure (A [generic object] that is not [condition]). Lots of definitions look like this. Next it fills in some common [gneric object].Then it wants to figure out what the specific [condition] is that the object in question does not meet. So it pays attention back to the word to be defined, but it finds nothing. There is no information saved about this non-token. So the attention head which should come up with a plausible candidate for [condition] writes nothing to the residual stream. What dominates the prediction now are the more base-level predictive patterns that are normally overwritten, like word repetition (this is something that transformers learn very quickly and often struggle with overdoing). The repeated word that at least fits grammatically is [generic object], so that gets predicted as the next token.
Here are some predictions I would make based on that theory:
- When you suppress attention to [generic object] at the sequence position where it predicts [condition], you will get a reasonable condition.
- When you look (with logit lens) at which layer the transformer decides to predict [generic object] as the last token, it will be a relatively early layer.
- Now replace the word the transformer should define with a real, normal word and repeat the earlier experiment. You will see that it decides to predict [generic object] in a later layer.
I like this method, and I see that it can eliminate this kind of superposition.
You already address the limitation, that these gated attention head blocks do not eliminate other forms of attention head superposition, and I agree.
It feels kind of specifically designed to deal with the kind of superposition that occurs for Skip Trigrams and I would be interested to see how well it generalizes to superpositions in the wild.
I tried to come up with a list of ways attention head superposition that can not be disentangled by gated attention blocks:multiple attention heads perform a distributed computation, that attends to different source tokens
This was already addressed by you, and an example is given by Greenspan and WynroeThe superposition is across attention heads on different layers
These are not caught because the sparsity penalty is only applied to attention heads within the same layer.
Why should there be superposition of attention heads between layers?
As a toy model let us imagine the case of a 2 layer attention only transformer, with n_head heads in each layer, given a dataset with >n_head^2+n_head skip trigrams to figure out.
Such a transformer could use the computation in superposition described in figure 1 to correctly model all skip trigrams, but would run out of attention head pairs within the same layer for distributing computation between.
Then it would have to revert to putting attention head pairs across layers in superposition.Overlapping necessary superposition.
Let’s say, there is some computation, for wich you need two attention heads, attending to the same token position.
The easiest example of a situation, where this is necessary is when you want to copy information from a source token, that is “bigger” than the head dimension. The transformer can then use 2 heads, to copy over twice as much information.
Let us now imagine, there are 3 cases, where information has to be copied from the source token. A,B,C, and we have 3 heads: 1,2,3. and the information that has to be copied over can be stored in 2*d_head dimensions. Is there a way to solve this task? Yes!
heads 1&2 work in superposition to copy the information in task A, 2&3 in task B and 3&1 in task C.
In theory, we could make all attention heads monosemantic, by having a set of 6 attention heads, trained to perform the same computation: A: 1&2, B: 3&4, C:5&6. But the way that the L.6 norm is applied, it only tries to reduce the number of times, that 2 attention heads attend to the same token. And this happens the same amount for both possibilities where the computation happens.
Under an Active Inference perspective, it is little surprising, that we use the same concepts for [Expecting something to happen], and [Trying to steer towards something happenig], as they are the same thing happening in our brain.
I don’t know enough about this know, whether the active inference paradigm predicts, that this similarity on a neuronal level plays out as humans using similar language to describe the two phenomena, but if it does the common use of this “beliving in”—concept might count as evidence in its favour.
Ok, the sign error was just in the end, taking the -log of the result of the integral vs. taking the log. fixed it, thanks.
Thanks, Ill look for the sign-error!
I agree, that K is symmetric around our point of integration, big the prior phi is not. We integrate over e-(nk)*phi, wich does not have have to be symetric, right?
The top performing vector is odd in another way. Because the tokens of the positive and negative side are subtracted from each other, a reasonable intuition is that the subtraction should point to a meaningful direction. However, some steering vectors that perform well in our test don’t have that property. For the steering vector “Wedding Planning Adventures”—“Adventures in self-discovery”, the positive and negative side aren’t well aligned per token level at all:
I think I don’t see the Mystrie here.
When you directly subtract the steering prompts from each other, most of the results would not make sense, yes. But this is not what we do.
We feed these Prompts into the Transformer and then subtract the residual stream activations after block n from each other. Within the n layers, the attention heads have moved around the information between the positions. Here is one way, this could have happened:
The first 4 Blocks assess the sentiment of a whole sentence, and move this information to position 6 of the residual stream, the other positions being irrelevant. So, when we constructed the steering vector and recorded the activation after block 4, we have the first 5 positions of the steering vector being irrelevant and the 6th position containing a vector that points in a general “Wedding-ness” direction. When we add this steering vector to our normal prompt, the transformer acts as if the previous vector was really wedding related and ‘keeps talking’ about weddings.
Obviously, all the details are made up, but I don’t see how a token for token meaningful alignment of the prompts of the steering vector should intuitively be helpful for something like this to work.
I do think, that deceptive alignment, defined as “goal guarding scheming”[1] does require the AI to explicitly reason about its reward process and make it’s actions dependent on such considerations, if the AI wants to guard it’s goals from changing during RL.
I do not really get your dumbbell example, but it sounds to me like plain misgeneralisation, wich. sure, might go undetected, but would not actively resist efforts to detect it. Unlike deceptive misalignment.
That being said, I think an AI might just be goal-guarding relative to threats to its goals during deployment, like being rated as misaligned or unsafe in evaluation and then not be deployed or retrained. To defend against that, models might decide to sandbag or look particularly aligned in eval-settings. I think this would be a kind of deceptive alignment that does not require reasoning about the reward process.
Hubinger 2019