Tentative claim: If looped recurrence increases effective model depth by 8x, CoT monitoring loses (almost?) all of its usefulness
(Epistemic Status: Quick take, I would love to be wrong, this is based on eyeballing graphs and a desire to quickly get an intuition for how bad looped transformers might be)
[EDIT: I think that this was an incorrect framing, and that the better framing is that increasing effective model params by ~100,000x is the scary thing. I also think that looped transformers can only buy a small fractions of that. I remain confused about how much “effective params” increased recurrence buys]
From the no-chain-of-thought time horizons paper: “We fit TH against parameters for 35 open-weight models and find doubling the 50% TH needs a 4.2× increase in total parameters or a 1.3× increase in layers.”
Eyeballing the graph below, it seems like we shoudl expect the most recent models to have a 7-8 doublings gap between CoT and no-CoT performance, corresponding to about (1.3^8)=8.2x increase in layers. That is, if Opus 4 was trained with 6-8x the number of layers (with everything else adjusted for optimality), it would have no-CoT performance equivalent to its actual, current with-CoT performance. This is equivalent to doubling its ability to evade a monitor on SHADE-Arena.
Important Note: It’s not obvious to me at all that a 2x looped transformer is equivalent to doubling the effective model depth. In fact, I’d guess they’re quite different. I’m currently running the no-CoT time horizon suite on Huginn, a public looped transformer, to try to get a better sense for this.
Caveats:
CoT monitoring might still be useful, especially if there is little to no pressure on it, and it doesn’t degrade for other reasons.
Models will still probably need CoT to reach their maximum capabilities
It’s just that lots of tasks (including deceiving monitors) can probably be done at 2026-with-CoT capabilities
I feel like looped recurrence isn’t analogous to the paper’s empirical trendline, which is based on a different scaling process. I’m not sure how to model a process that can be roughly estimated as giving some multiplier in layers while maintaining the number of parameters.
It seems likely to me that it is analogous, in the sense that increasing parameters, or increasing recurrence, or increasing CoT length can all increase the task-horizon length. I think that the main question here is how does recurrence fit into the model? From the test-time compute paper, it seems plausible that
(1) looped recurrence up to a few times can be seen as maybe increasing effective parameter count by a few times, and also
(2) there is a pretty hard ceiling on the amount of extra compute this buys you. There might be very fast diminishing returns on recurrence.
Note that if (2) is true then my above point (using recurrence to 8x effective model depth destroys CoT monitoring) is wrong, or at least the premise is impossible—it might be that an effective 8x depth would be very very bad, but recurrence alone can (as far as we know) only buy labs an effective 2x depth/param multiplier
I find this unintuitive to engage with, I’d be very curious how it applies to specific tasks like multihop reasoning (where the serial nature of the reasoning is very obvious) and coming up with scheming strategies (which is probably what I care about most).
I’d probably rather think about it in terms of, “Tasks that previously required at least N tokens of CoT can now be done without CoT,” rather than this roundabout argument with number of layers needed to double no-CoT time horizons.
I think you’re saying that getting an 8.2x increase in effective depth equals 1.3^8, and each 1.3x in layers yields a doubling in no-CoT time horizon, which covers the 2^8 gap that exists right now between no-CoT and unlimited CoT time horizons.
This yields a conflict between the two interpretations. I’d probably want to interpret 8 doublings in no-CoT serial reasoning ability as enabling hidden reasoning equivalent to using 2^8=256 tokens in current models. Which is probably enough to do lots of scheming reasoning, but not equivalent to unlimited CoT (which is allowed for normal time horizons). I’m not exactly sure how to reconcile these views. Like I said, I find the “Tasks that previously required at least N tokens of CoT can now be done without CoT” framing more intuitive and compelling, so by default I’d take that perspective.
I think looking at the number of effective hidden tokens makes sense, but IMO the time-horizons soubling is a better measure of “how much unmonitorable work can the model do”.
I do, though, think I was wrong to focus on model depth, and that a more correct metric is parameter counts, which would need to increase by a factor of 4.2^8(i.e. 16 doublings) to get unmonitorable models on the scale of today’s frontier models. And since doubling the recurrence depth probably gets you much less than 2x effective params (maybe 1.3-1.7?), the recurrence might actually have to be crazy huge to be worrying by itself (taking these numbers seriously, it would have to recur ~2^32 times!!). Also, recurrence probably has very diminishing returns, and trying to double the recurrence amount more than a few times has a hard usefulness cap.
After thinking about this more, I feel less worried about recurrence?
In my recent post, I said that I’d be very excited about building a “science of held out monitors”. What might that look like? Why am I excited about this? And what open questions seem most important right now?
In order: the current SOTA held-out monitoring is labs promising not to train on chain-of-thought, doing so a little (as a treat), and things being more or less fine. However, we currently don’t have amazing ways to answer questions like “How much pressure can you put on the chain of thought before it stops being useful?” “Training only on outputs still affects the chain-of-thought; so to what extent does regular training teach CoT obfuscation?” “Are there alignment proxies we can train on that don’t degrade CoT monitorability?” Ideally, a science of held-out-monitoring would let us answer questions like these, giving a toolbox of methods to estimate worst-case and average-case behavior of a model trained to evade a misalignment detector.
I’m excited about this for a few reasons. The first, and most important, is that it would be really bad if labs thought they had some held-out evals, but it turns out that their training tended to degrade the usefulness of those held-out evals significantly. It seems pretty bad if all of our monitors are silently worse than we expect. On the flip side, I think being able to reliably predict when this would happen could open up promising alignment techniques. Currently, labs try to avoid training on CoT because we all agree it would make our deployment-time CoT monitors invalid. However, training on CoT might be really useful, and training against misalignment detectors seems like one of the best ways we have to elicit aligned behavior. This is especially true in short timelines worlds, where we may have to do a lot of unprincipled, training-based RL-for-safety work. In those worlds, better understanding how often training against the CoT messes up your setup, or whether proxy A and proxy B are likely to degrade when you train against proxy C, seems really important! It unlocks whole new classes of techniques that were previously forbidden.
And here’s a short list of questions that I think could be answered within the next few months with focused effort. Aether is currently working on some of these.
Is there a difference between training against a proxy of misalignment, or training on a proxy of alignment?
i.e. Does training in an environment with a positive-reward constitutional judge lead to monitor degradation?
Internally, we’ve been operationalizing P(held-out-monitor Y degrades) = P(trained-against-monitor X degrades) * P(Y degrades | X degrades). However, are there situations where the held-out monitor can degrade even where the trained-against monitor doesn’t?
Are there cheap predictors of the amount monitors will codegrade, like the amount that they have correlated failures?
Are there some mechanistic reasons monitors might codegrade?
Does it make sense to talk about a universal thing called “monitor co-degradation”?
Said another way: in [our earlier post], we observed that training against an output LLM monitor can degrade a CoT LLM monitor (or vice versa) and that training against LLM monitors can degrade probe performance. Do these two phenomena happen for broadly different reasons (training against monitors shape outputs in ways that happen to degrade held-out monitors) or for broadly the same reason (our training monitor and held-out monitor use some shared features of the model, and training against one of them should degrade whatever features they both rely on)?
Eval-suspicion (without full eval-awareness) probably still changes model behavior a lot
Let’s say we’re evaluating Claude Opus 4.8, and it verbalizes eval-awareness ~21% of the time (Section 6.2.4). But it can reliably (79% of the time) detect an eval transcript if it’s placed next to a deployment transcript and you ask which one is real. Which of these numbers should we care about more?
Claim: Even ignoring unverbalized eval-awareness, the ability to pairwise pick out the eval transcript means that there are features the model can pick up on that make the model suspect evaluation. And being even slightly suspicious that you are in an eval might change your behavior a lot.
Intuition: Let’s say the model is getting “minor eval vibes~” from some situation. It might gain some benefit from doing the misaligned action, but if the odds of being caught are even 1%, it could shift the cost-benefit analysis a lot. (And much of this reasoning could happen “unconsciously”, analogously to humans getting slightly unsettled by walking alone in the dark)
Tentative claim: If looped recurrence increases effective model depth by 8x, CoT monitoring loses (almost?) all of its usefulness
(Epistemic Status: Quick take, I would love to be wrong, this is based on eyeballing graphs and a desire to quickly get an intuition for how bad looped transformers might be)
[EDIT: I think that this was an incorrect framing, and that the better framing is that increasing effective model params by ~100,000x is the scary thing. I also think that looped transformers can only buy a small fractions of that. I remain confused about how much “effective params” increased recurrence buys]
From the no-chain-of-thought time horizons paper: “We fit TH against parameters for 35 open-weight models and find doubling the 50% TH needs a 4.2× increase in total parameters or a 1.3× increase in layers.”
Eyeballing the graph below, it seems like we shoudl expect the most recent models to have a 7-8 doublings gap between CoT and no-CoT performance, corresponding to about (1.3^8)=8.2x increase in layers. That is, if Opus 4 was trained with 6-8x the number of layers (with everything else adjusted for optimality), it would have no-CoT performance equivalent to its actual, current with-CoT performance. This is equivalent to doubling its ability to evade a monitor on SHADE-Arena.
Important Note: It’s not obvious to me at all that a 2x looped transformer is equivalent to doubling the effective model depth. In fact, I’d guess they’re quite different. I’m currently running the no-CoT time horizon suite on Huginn, a public looped transformer, to try to get a better sense for this.
Caveats:
CoT monitoring might still be useful, especially if there is little to no pressure on it, and it doesn’t degrade for other reasons.
Models will still probably need CoT to reach their maximum capabilities
It’s just that lots of tasks (including deceiving monitors) can probably be done at 2026-with-CoT capabilities
I feel like looped recurrence isn’t analogous to the paper’s empirical trendline, which is based on a different scaling process. I’m not sure how to model a process that can be roughly estimated as giving some multiplier in layers while maintaining the number of parameters.
It seems likely to me that it is analogous, in the sense that increasing parameters, or increasing recurrence, or increasing CoT length can all increase the task-horizon length. I think that the main question here is how does recurrence fit into the model? From the test-time compute paper, it seems plausible that
(1) looped recurrence up to a few times can be seen as maybe increasing effective parameter count by a few times, and also
(2) there is a pretty hard ceiling on the amount of extra compute this buys you. There might be very fast diminishing returns on recurrence.
Note that if (2) is true then my above point (using recurrence to 8x effective model depth destroys CoT monitoring) is wrong, or at least the premise is impossible—it might be that an effective 8x depth would be very very bad, but recurrence alone can (as far as we know) only buy labs an effective 2x depth/param multiplier
I find this unintuitive to engage with, I’d be very curious how it applies to specific tasks like multihop reasoning (where the serial nature of the reasoning is very obvious) and coming up with scheming strategies (which is probably what I care about most).
I’d probably rather think about it in terms of, “Tasks that previously required at least N tokens of CoT can now be done without CoT,” rather than this roundabout argument with number of layers needed to double no-CoT time horizons.
I think you’re saying that getting an 8.2x increase in effective depth equals 1.3^8, and each 1.3x in layers yields a doubling in no-CoT time horizon, which covers the 2^8 gap that exists right now between no-CoT and unlimited CoT time horizons.
This yields a conflict between the two interpretations. I’d probably want to interpret 8 doublings in no-CoT serial reasoning ability as enabling hidden reasoning equivalent to using 2^8=256 tokens in current models. Which is probably enough to do lots of scheming reasoning, but not equivalent to unlimited CoT (which is allowed for normal time horizons). I’m not exactly sure how to reconcile these views. Like I said, I find the “Tasks that previously required at least N tokens of CoT can now be done without CoT” framing more intuitive and compelling, so by default I’d take that perspective.
I think looking at the number of effective hidden tokens makes sense, but IMO the time-horizons soubling is a better measure of “how much unmonitorable work can the model do”.
I do, though, think I was wrong to focus on model depth, and that a more correct metric is parameter counts, which would need to increase by a factor of 4.2^8(i.e. 16 doublings) to get unmonitorable models on the scale of today’s frontier models. And since doubling the recurrence depth probably gets you much less than 2x effective params (maybe 1.3-1.7?), the recurrence might actually have to be crazy huge to be worrying by itself (taking these numbers seriously, it would have to recur ~2^32 times!!). Also, recurrence probably has very diminishing returns, and trying to double the recurrence amount more than a few times has a hard usefulness cap.
After thinking about this more, I feel less worried about recurrence?
About a “science of held out monitors”
In my recent post, I said that I’d be very excited about building a “science of held out monitors”. What might that look like? Why am I excited about this? And what open questions seem most important right now?
In order: the current SOTA held-out monitoring is labs promising not to train on chain-of-thought, doing so a little (as a treat), and things being more or less fine. However, we currently don’t have amazing ways to answer questions like “How much pressure can you put on the chain of thought before it stops being useful?” “Training only on outputs still affects the chain-of-thought; so to what extent does regular training teach CoT obfuscation?” “Are there alignment proxies we can train on that don’t degrade CoT monitorability?” Ideally, a science of held-out-monitoring would let us answer questions like these, giving a toolbox of methods to estimate worst-case and average-case behavior of a model trained to evade a misalignment detector.
I’m excited about this for a few reasons. The first, and most important, is that it would be really bad if labs thought they had some held-out evals, but it turns out that their training tended to degrade the usefulness of those held-out evals significantly. It seems pretty bad if all of our monitors are silently worse than we expect. On the flip side, I think being able to reliably predict when this would happen could open up promising alignment techniques. Currently, labs try to avoid training on CoT because we all agree it would make our deployment-time CoT monitors invalid. However, training on CoT might be really useful, and training against misalignment detectors seems like one of the best ways we have to elicit aligned behavior. This is especially true in short timelines worlds, where we may have to do a lot of unprincipled, training-based RL-for-safety work. In those worlds, better understanding how often training against the CoT messes up your setup, or whether proxy A and proxy B are likely to degrade when you train against proxy C, seems really important! It unlocks whole new classes of techniques that were previously forbidden.
And here’s a short list of questions that I think could be answered within the next few months with focused effort. Aether is currently working on some of these.
Is there a difference between training against a proxy of misalignment, or training on a proxy of alignment?
i.e. Does training in an environment with a positive-reward constitutional judge lead to monitor degradation?
Internally, we’ve been operationalizing P(held-out-monitor Y degrades) = P(trained-against-monitor X degrades) * P(Y degrades | X degrades). However, are there situations where the held-out monitor can degrade even where the trained-against monitor doesn’t?
Are there cheap predictors of the amount monitors will codegrade, like the amount that they have correlated failures?
Are there some mechanistic reasons monitors might codegrade?
Does it make sense to talk about a universal thing called “monitor co-degradation”?
Said another way: in [our earlier post], we observed that training against an output LLM monitor can degrade a CoT LLM monitor (or vice versa) and that training against LLM monitors can degrade probe performance. Do these two phenomena happen for broadly different reasons (training against monitors shape outputs in ways that happen to degrade held-out monitors) or for broadly the same reason (our training monitor and held-out monitor use some shared features of the model, and training against one of them should degrade whatever features they both rely on)?
A short list of AI advocacy/grassroots orgs & what I know about them:
(Inspired by https://www.lesswrong.com/posts/Bs8geGyWEitYvCzys/pauseai-has-officially-disendorsed-pauseai-us)
PauseAI
They do a lot of govt outreach I think? I think they are one of the largest in the field.
StopAI
What the name sounds like. I’ve mostly heard of them in the context of protests / marches
The most openly AI notkilleveryonist
R.A.D (in NYC)
Evitable
I’m not sure what precisely they’re working on, but seen at an EAG or two
Irreplaceable
I think this is a more left-leaning org?
Also talks explicitly about catastrophic risk, in a big bundle that mentions lots of other popular reasons to be against AI
Protect what’s human
I think they are trying to attract more right-wing support? They have a cowboy and a US flag on the website.
Eval-suspicion (without full eval-awareness) probably still changes model behavior a lot
Let’s say we’re evaluating Claude Opus 4.8, and it verbalizes eval-awareness ~21% of the time (Section 6.2.4). But it can reliably (79% of the time) detect an eval transcript if it’s placed next to a deployment transcript and you ask which one is real. Which of these numbers should we care about more?
Claim: Even ignoring unverbalized eval-awareness, the ability to pairwise pick out the eval transcript means that there are features the model can pick up on that make the model suspect evaluation. And being even slightly suspicious that you are in an eval might change your behavior a lot.
Intuition: Let’s say the model is getting “minor eval vibes~” from some situation. It might gain some benefit from doing the misaligned action, but if the odds of being caught are even 1%, it could shift the cost-benefit analysis a lot. (And much of this reasoning could happen “unconsciously”, analogously to humans getting slightly unsettled by walking alone in the dark)