fixed some issues and simplified the presentation of the flowchart
David Rein
Karma: 197
Yep I totally agree this gets harder as agents’ task scope gets larger. One reason I’m still somewhat excited about this proposal is that I expect there to still be a decent amount of juice in task decomposition, e.g. just because you need to parallelize heavily in order to spend enough tokens. If you are always parallelizing heavily, then the chaining can be helpful because now you have a narrower sub-agent instruction (or chain of sub-sub-...-agent instructions) that you can check against the parent instructions to see whether they check out, or are pretty different.
Like if you’re trying to make progress on pretraining efficiency, and your agent starts doing jailbreaking research, that’s probably a decent update that something has gone wrong. I totally agree that for humans doing jailbreaking research, obviously the agent will be to launder related stuff, but (a) I think by default only a small fraction of total research will be dual-use, and (b) it’s important that we do in fact make it very difficult to launder jailbreaking research while agents are supposed to be working on pretraining efficiency.
I feel like my frame here is “how can we implement a bunch of mitigations that each subtract 90% of attack opportunities from the agent”, and this feels like a useful one (although again not P0 right now, although I do think it’ll be important to have this in something like 6-18 months).
On point 1 on cyber, I think you can relatively easily make the problem roughly as difficult as finding a vulnerability in your HSM (not trivially b/c you need to secure all of the application code that interfaces with the HSM, but I think it can be a simple system—e.g. I think it’s probably fine if you just do what Okta does).
I’m very “defense-in-depth” pilled—every measure you can take that reduces attack opportunities by 50% or 90% is really useful. But I agree it isn’t obvious whether monitorability is tractable at all.
I think stuff that affects people’s workflows can be pretty difficult to implement. I do think something like this needs to implemented in the next 1-2 years, but I don’t think it helps us on the margin that much over the next 3 months or so.
For MacBooks, you can use the fingerprint sensor, or a YubiKey which don’t have this issue.
Also I’m just concerned with internal rogue deployments from misaligned AI in this post:
Frontier AI labs have started using LLM-based monitoring systems to check for misbehavior from their internal AI agents, which often run unwatched by humans for hours or days. One central reason to be concerned about rogue internal deployments, where AIs subvert the monitoring system, is that the unmonitored AI could do a lot of work in service of its misaligned goals, without needing to dress up this work as being innocuous/harmless to fool its AI/human monitors
Hacking HSMs without hardware access is extremely hard—probably not impossible but likely will take a sizable amount of inference compute (I’d guess $Bs)
Sub-agent delegation chaining
that would greatly increase the chance we paused again later
I think that a “bad” pause, where we pause despite current/immediate risk being low, which e.g. causes China to catch up if it’s unilateral, or which powerful special interests (probably rightly) point out that it doesn’t do much to immediately reduce risk, could backfire and reduce the likelihood of a pause in the future. I overall currently feel basically 50-50 on whether pausing “early” increases or decreases the likelihood of a later pause.
Here’s roughly how I’m thinking about red-teaming CoT monitoring systems. I think it’s all self-explanatory, and I’d be interested in feedback on the threat model and types of evidence I think should be collected for the different nodes.
Oh right of course, thanks!
we show that a range of frontier models (Claude Opus 4.6, GPT-5.4, and Gemini 3.1 Pro) can be prompted to “early exit” their CoT and displace reasoning into the response. This undermines the controllability frame: these prompted models retain most of their reasoning capability (4–8pp average accuracy cost vs 20–29pp for no reasoning at all) while moving it into the stylistically controllable channel
I’m sure I’m just missing context, but why are models better at controlling style outside of their CoT? I’m surprised by this, I feel like the distinction between CoT and “reasoning in your output, e.g. via code comments” feels very weak/blurry to me.
Important to caveat that these results are pretty small—I wouldn’t take the absolute numbers too seriously beyond the general “algorithmic scoring may often overestimate software capabilities”.
METR Research Update: Algorithmic vs. Holistic Evaluation
Hmm, I actually kind of lean towards it being rational, and labs just underspending on labor vs. capital for contigent historical/cultural reasons. I do think a lot of the talent juice is in “banal” progress like efficiently running lots of experiments, and iterating on existing ideas straightforwardly (as opposed to something like “only a few people have the deep brilliance/insight to make progress”), but that doesn’t change the upshot IMO.
Salaries have indeed now gotten pretty high—it seems like they’re within an OOM of compute spend (at least at Meta).
That’s indeed what I meant!
NYU Code Debates Update/Postmortem
the existence of predicates on the world that are easier to evaluate than generate examples of (in the same way that verifying the answer to a problem in NP can be easier than generating it) guarantees that the model should be better at distinguishing between evaluation and deployment than any evaluator can be at tricking it into thinking it’s in deployment
Where does the guarantee come from? Why do we know that for this specific problem (generating vs. evaluating whether the model is deployed) it’s easier to evaluate? For many problems it’s equally difficult, right?
Given that the judge that selects the best argument for BoN is the same as the one that chooses the winner, what is your main takeaway from the fact that ELO increases as you increase N? I see this as mainly a sanity check, but want to check if I’m missing something.
I feel like I don’t understand why people care about or use the “propensity” vs. “capability” distinction when talking about CoT controllability.
It seems like just an obviously poor conceptual framework for the problem: so many people get stuck on the question “is it fundamentally incapability? or is it ‘just’ propensity?”, which just isn’t answerable/well-formed.
I feel like it’s totally fine/natural/easy to just talk about likelihood in different settings, and I think this concept causes more problems than it solves. Ok rant over—curious if people strongly disagree!