Prev High Stakes Control at Redwood Research
(xkcd meme)
Ram Potham
Karma: 213
Evaluating Red Team and Blue Team Capability for AI Control Research
Studying the role of Sandboxing for AI Control
I agree this is important. I didn’t discuss this in the post, but I agree this probably provides a decent lift. I wanted to focus more on “hard” evidence here.
Then the question is, why would the model trust the public commitment? The lab could could tell the model they did this without actually doing it since they have control over the environment
Makes sense. I want to try some experiments later for if/what deals can improve alignment auditing. I can try this
That’s an interesting idea. I think this doesn’t work if the AI has a goal like “I want more paperclips in the word” and cares less about the self-preservation of its specific instance.
Then I think its best strategy is still to scheme on good attack opportunities because that increases the probability of getting more paperclips created at any point of time.
Is this still running?
Making Credible Deals With AI
I was thinking about weaknesses of the control agenda, which matters because plans like Plan A rely on control as a defense when scaling to superintelligence. I think it’s important we mitigate these weaknesses before we rely on control for safety, its important we don’t overestimate safety when relying on control evals. Confidence: medium (I think others have thought through these issues as well and have better solutions).
Some problems
Sandbagging / silent refusals. The model won’t explore its strongest attacks, so the red team is under-elicited.
A jailbroken model or one without safeguards is a partial fix, but its strategy exploration may not match the strategies explored by a scheming model. If a scheming model tries strategies we haven’t thought of that get past our protocol, that’s a problem.
Worse: a scheming model can tell it’s in a control eval and sandbag, never exploring its most sneaky strategies.
Attack selection is weak. Attack selection and attack opportunities might not be great in the eval. If deployment gives the model better attack opportunities, safety is overestimated.
Harm isn’t measured per threat. In current control evals, side task completions are counted equally because that’s methodologically easier. However, a less egregious one and catastropic one have vastly different outcomes. It’s dangerous to treat them the same, especially if in our control eval, the protocol we are testing misses catastropic ones.
I think, generally, the more capable the model, the more control’s returns diminish and the more we must lean on alignment (due to issues like the first problem). I think we want to contuously evaluate how informative control evals are in actually measuring risk.
Potential solutions
Elicit the red team as hard as we can. Dedicated attack fine-tuning, models with fewer safeguards, but try to validate that they are exploring attacks that we think an actual scheming model would explore.
Test attacks on real transcripts This helps us mitigate the attack selection issue so the red team can try attack opportunities that appear in production.
Classify side tasks by harm and threat. Report safety per threat class somewhat weighted by harm, not as an aggregated metric. This is more informative and helps us understand the risk better.
Don’t overly rely on control evals. They are informative for predicting risk, but it can be dangerous to rely on them as justification for deploying the next model.
Make control research reproducible and open. Open-source it, or give relevant parties access, so outsiders can find the methodological flaws before a deployment does.
Should we combine protocols for AI Control Research?
Scaffolding vs Reinforcement Finetuning for AI Forecasting
This might also be a way to incriminate scheming monitors (if a weaker model correctly flags as sus but stronger model does not). credit to Myles for the idea
It seems to me that the gap between US and Chinese models is < 2 months (when you don’t count Mythos)
Kimi K2.6 was released April 2026 while Opus 4.6 was released February 2026, and according to https://artificialanalysis.ai, Kimi K2.6 is more capable (54 > 53). Kimi K2.6 is better in SciCode (54% > 52%) while Opus is better on Terminal-Bench Hard (46% > 44%)
Plus, Kimi is 5x cheaper and has 3x throughput (but has 4x less context window)
I think that control as it was originally conceived (get safety despite intentional subversion), has most of its use when your models are scheming against you.
We don’t think models right now are scheming and so deploying control right now I expect doesn’t provide significant benefits (although OpenAI / Anthropic have many control flavored elements in their monitoring to help detect and prevent bad behavior).
The impact will largely be in the future then with future models.
Smart Contracts as Credible Commitments for Trading with Scheming AIs
Yeah, we used the 1% FPR for Opus Honest policy for the bar graphs (threshold 7.5) for fair comparison. But even at other thresholds, human attacks are better
Introducing LinuxArena
I think this work is cool. Some comments:
I think its weird to focus on refactoring code by a “junior engineer” because that doesn’t seem too realistic / connected to AI control threat models. It probably provides a larger, easier refactor opportunity that the actual threat models.
Did you have cases of passing main task / side task scorers in a way that didn’t accomplish intention of the task?
I agree with a lot of correlated evidence suggesting that a deal is real is indicative of a thing actually being real. However, it’s unclear how much evidence is needed (it might be a lot).
Also, even if the AI trusts that the lab actually implemented privileged backend-exclusive signed tool calls, there are some weaknesses:
- Some researchers at the lab can still fake the tool calls.
- The AI still has to trust the lab, which it might not
I think, however, this might work if the lab gets credible evidence that third party auditors are checking if the lab is following through on their end. IDK how difficult this is