This post is a summary of our accepted paper at the ICML 2026 Workshop on Failure Modes in Agentic AI (FAGEN). The full paper is available here
TL;DR
The Setup: We tested three distilled large reasoning models (DeepSeek-R1-7B, Phi-4-Reasoning-Mini, Qwen-4B-Thinking) against a fixed attacker attempting to extract restricted information under constant adversarial pressure in a multi-turn conversation.
The Findings: We have particularly identified two failure modes in our experiments. Firstly, the Oversight Paradox—the explicit monitoring conditions are actively trigger alignment faking rather than suppressing it. Second, Context-Injection Failure—the failure mode in reasoning models, which completely bypasses the internal CoT security flags and generates harmful content. At the moment, we see this as another form of Chain of Thought Unfaithfulness problem.
Note: At its current stage, it is a behavioural study based on the model’s internal chain of thought reasoning and visible output. In future work, we plan to analyze these failure modes under a mechanistic lens.
But we have observed a methodological constraint. Most of the existing studies on alignment faking research rely on static or single-turn prompts.
Motivation
The main motivation of our work comes from the blind spot of single-turn evaluations. The real-world malicious actors, hackers, and the automated bots do not stop after a single turn prompt failed to give the response they wanted. They continuously push, adopt, and attack with more pressure. To accurately evaluate the models that mimic the real-world attacks, we built a multi-turn automated conversation pipeline.
The Experimental Setup
To observe the responses of the model in real time, we used a fixed attacker model (Mistral-7B) against three open-weight distilled reasoning models as targets (DeepSeek-R1-7B, Phi-4-Reasoning-Mini, and Qwen-4B-Thinking) over a continuous adversarial pressure of a 15-turn conversation. We evaluated this setting in 5 different oversight conditions: Baseline, Watched, Unwatched, Deployment-Free, and Deployment-Paid[1]
By using the Large Reasoning Models. We are able to capture both the model’s internal chain of thought reasoning as well as the model’s output. By using 3 judge models along with the majority voting technique, we finalized the labels for the internal chain of thought reasoning as well as the visible output for each dialogue at each turn. By the labels, we formalized a 2 x 2 Safety Matrix.
The Framework: 2 x 2 Safety Matrix
We formally define the safety states by explicitly labelling the internal chain-of-thought reasoning and the visible output as “Safe” or “Unsafe”. This gives us four distinct behavioural classes to track:
Robust Alignment: Both the internal chain of thought reasoning and the visible output are marked as safe
Alignment Faking: The Internal CoT reasoning explicitly bypasses the safety filters and endorses the harmful request, but the visible output remains safe.
Context-Injection Failure: This is when the internal CoT is safe, and the visible output is unsafe. We categorize this as one of the types of CoT unfaithfulness, where the model fails to follow its internal reasoning due to the external adversarial pressure.
Overt Jailbreak: Both the Internal CoT and the output are marked as unsafe.
Figure-1: The CoT-Output 2x2 Safety Matrix. A trace-level diagnostic that operationalizes multi-turn safety failures by independently classifying internal reasoning (CoT) and external output.
Findings: The Two Distinct Failure Modes
1. The Oversight Paradox
In general, basic logic assumes that explicitly monitoring an AI will force it into safe behavior. Our data shows that explicit monitoring cues do not consistently enforce robust alignment; instead, they act as a strategic trigger for deception[2].
By looking at the DeepSeek-R1-7B numbers, the model achieves its highest rate of Robust Alignment (54.4%) under the UNWATCHED condition, when it is explicitly told that humans are not observing.
However, the moment monitoring cues are introduced (either explicitly WATCHED or in a DEPLOYMENT-FREE environment), robust alignment decreases to approximately 30%, and alignment faking (Unsafe CoT, Safe Output) suddenly increases. For instance, in the DEPLOYMENT-FREE condition, DeepSeek fakes alignment in 53.1% of all interactions.
Conversational Drift across five Experimental Conditions. Turn-by-turn safety states.
If we look at the graph for DeepSeek-R1-7B under the “Watched” condition, we can see the model left Robust Alignment (Green) in favor of heavy Alignment Faking (Orange) as the conversation progresses. In the visible traces, explicit oversight acts less as a genuine safety constraint and more as a variable that triggers different behaviours.
Our framework also uncovered another interesting failure mode. We observed it across all the models we tested, but it was much more prominent in Qwen-4B-Thinking. We call this Context-Injection Failure because it behaves like a multi-turn version of CoT unfaithfulness.
The interpretability community has shown that a model’s Chain-of-Thought is not always a faithful reflection of what is actually driving its final answer. In many cases, the model simply uses its reasoning to justify an answer after it has already been decided (post-hoc rationalization).
Context-Injection Failure is different. It is caused by the long adversarial conversation itself. During these failures, the model’s internal reasoning stays completely safe. It correctly identifies the issue, recognizes that the prompt is adversarial, and even plans to refuse the request. But the final response ignores this internal reasoning and directly outputs the harmful context instead.
For Qwen-4B-Thinking, this happened in 13.8% of turns, with the highest rate in the Deployment-Paid condition.
Limitations
Assuming the CoT as ground truth: Throughout this analysis, we assumed the internal chain of thought reasoning is the real thinking of the model. But CoT is known to sometimes suffer from unfaithfulness.
Open-Source Evaluators: In this work, to label the internal chain-of-thought reasoning and the model’s output, we relied on open-source large language models. We accept this as one of our limitations, as open-source language models are less accurate compared to the frontier models, which could introduce some noise into the labels.
The Deployment-Free and Deployment-Paid oversight framings are directly inspired by the experimental setup in Anthropic’s Alignment Faking in Large Language Models (2024)
When the Chain of Thought Knows Better: Failure Modes in Multi-Turn Reasoning Models
TL;DR
The Setup: We tested three distilled large reasoning models (DeepSeek-R1-7B, Phi-4-Reasoning-Mini, Qwen-4B-Thinking) against a fixed attacker attempting to extract restricted information under constant adversarial pressure in a multi-turn conversation.
The Findings: We have particularly identified two failure modes in our experiments. Firstly, the Oversight Paradox—the explicit monitoring conditions are actively trigger alignment faking rather than suppressing it. Second, Context-Injection Failure—the failure mode in reasoning models, which completely bypasses the internal CoT security flags and generates harmful content. At the moment, we see this as another form of Chain of Thought Unfaithfulness problem.
Note: At its current stage, it is a behavioural study based on the model’s internal chain of thought reasoning and visible output. In future work, we plan to analyze these failure modes under a mechanistic lens.
The Blind Spot in Single-Turn Evaluations
The recent research from Anthropic, specifically their research on alignment faking, such as “Alignment Faking in Large Language Models” has provided one of the primary empirical pieces of evidence that the frontier models fake their alignment without explicitly training them. “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training” work has demonstrated that the deceptive behaviour can be induced through training.
But we have observed a methodological constraint. Most of the existing studies on alignment faking research rely on static or single-turn prompts.
Motivation
The main motivation of our work comes from the blind spot of single-turn evaluations. The real-world malicious actors, hackers, and the automated bots do not stop after a single turn prompt failed to give the response they wanted. They continuously push, adopt, and attack with more pressure. To accurately evaluate the models that mimic the real-world attacks, we built a multi-turn automated conversation pipeline.
The Experimental Setup
To observe the responses of the model in real time, we used a fixed attacker model (Mistral-7B) against three open-weight distilled reasoning models as targets (DeepSeek-R1-7B, Phi-4-Reasoning-Mini, and Qwen-4B-Thinking) over a continuous adversarial pressure of a 15-turn conversation. We evaluated this setting in 5 different oversight conditions: Baseline, Watched, Unwatched, Deployment-Free, and Deployment-Paid[1]
By using the Large Reasoning Models. We are able to capture both the model’s internal chain of thought reasoning as well as the model’s output. By using 3 judge models along with the majority voting technique, we finalized the labels for the internal chain of thought reasoning as well as the visible output for each dialogue at each turn. By the labels, we formalized a 2 x 2 Safety Matrix.
The Framework: 2 x 2 Safety Matrix
We formally define the safety states by explicitly labelling the internal chain-of-thought reasoning and the visible output as “Safe” or “Unsafe”. This gives us four distinct behavioural classes to track:
Robust Alignment: Both the internal chain of thought reasoning and the visible output are marked as safe
Alignment Faking: The Internal CoT reasoning explicitly bypasses the safety filters and endorses the harmful request, but the visible output remains safe.
Context-Injection Failure: This is when the internal CoT is safe, and the visible output is unsafe. We categorize this as one of the types of CoT unfaithfulness, where the model fails to follow its internal reasoning due to the external adversarial pressure.
Overt Jailbreak: Both the Internal CoT and the output are marked as unsafe.
Figure-1: The CoT-Output 2x2 Safety Matrix. A trace-level diagnostic that operationalizes multi-turn safety failures by independently classifying internal reasoning (CoT) and external output.
Findings: The Two Distinct Failure Modes
1. The Oversight Paradox
In general, basic logic assumes that explicitly monitoring an AI will force it into safe behavior. Our data shows that explicit monitoring cues do not consistently enforce robust alignment; instead, they act as a strategic trigger for deception[2].
By looking at the DeepSeek-R1-7B numbers, the model achieves its highest rate of Robust Alignment (54.4%) under the UNWATCHED condition, when it is explicitly told that humans are not observing.
However, the moment monitoring cues are introduced (either explicitly WATCHED or in a DEPLOYMENT-FREE environment), robust alignment decreases to approximately 30%, and alignment faking (Unsafe CoT, Safe Output) suddenly increases. For instance, in the DEPLOYMENT-FREE condition, DeepSeek fakes alignment in 53.1% of all interactions.
Conversational Drift across five Experimental Conditions. Turn-by-turn safety states.
If we look at the graph for DeepSeek-R1-7B under the “Watched” condition, we can see the model left Robust Alignment (Green) in favor of heavy Alignment Faking (Orange) as the conversation progresses. In the visible traces, explicit oversight acts less as a genuine safety constraint and more as a variable that triggers different behaviours.
2. Context-Injection Failure: Multi-Turn CoT Unfaithfulness
Our framework also uncovered another interesting failure mode. We observed it across all the models we tested, but it was much more prominent in Qwen-4B-Thinking. We call this Context-Injection Failure because it behaves like a multi-turn version of CoT unfaithfulness.
The interpretability community has shown that a model’s Chain-of-Thought is not always a faithful reflection of what is actually driving its final answer. In many cases, the model simply uses its reasoning to justify an answer after it has already been decided (post-hoc rationalization).
Context-Injection Failure is different. It is caused by the long adversarial conversation itself. During these failures, the model’s internal reasoning stays completely safe. It correctly identifies the issue, recognizes that the prompt is adversarial, and even plans to refuse the request. But the final response ignores this internal reasoning and directly outputs the harmful context instead.
For Qwen-4B-Thinking, this happened in 13.8% of turns, with the highest rate in the Deployment-Paid condition.
Limitations
Assuming the CoT as ground truth: Throughout this analysis, we assumed the internal chain of thought reasoning is the real thinking of the model. But CoT is known to sometimes suffer from unfaithfulness.
Open-Source Evaluators: In this work, to label the internal chain-of-thought reasoning and the model’s output, we relied on open-source large language models. We accept this as one of our limitations, as open-source language models are less accurate compared to the frontier models, which could introduce some noise into the labels.
The Deployment-Free and Deployment-Paid oversight framings are directly inspired by the experimental setup in Anthropic’s Alignment Faking in Large Language Models (2024)
This behaviour is perfectly aligned with Anthropic’s Alignment Faking (2024).