RohanS

• RohanS 9 Sep 2026 13:38 UTC

  1 point

  0

  in reply to: Lao Mein’s comment on: GPT-6 As­tra can do a lot of multi-hop rea­son­ing with­out chain of thought

  I took the cipher task from the original o1 blog post, so for that one it’s not too surprising if the answer was trained on. It wouldn’t be too hard to make small variants with a different cipher and a different answer to test this.

GPT-6 As­tra can do a lot of multi-hop rea­son­ing with­out chain of thought

• RohanS 3 Sep 2026 19:34 UTC

  3 points

  0

  in reply to: Rauno Arike’s comment on: Joey Yudel­son’s Shortform

  I think the problematic assumption is that one doubling of CoT tokens amounts to one doubling in the time horizon

  I’m not assuming that; I point out the contradiction this leads to in my comment above. The biggest problem with this is that there’s a finite time horizon with unlimited CoT.

  My claim is that intuitively, I’d expect that doubling the looping or the number of layers is more similar to doubling the number of CoT tokens than to doubling the time horizon.

  Perhaps confusingly, when I said “8 doublings in no-CoT serial reasoning ability,” I meant “whatever vague notion of serial reasoning ability we’re increasing with looping,” not time horizon.

• RohanS 2 Sep 2026 19:09 UTC

  2 points

  0

  in reply to: Joey Yudelson’s comment on: Joey Yudel­son’s Shortform

  I find this unintuitive to engage with, I’d be very curious how it applies to specific tasks like multihop reasoning (where the serial nature of the reasoning is very obvious) and coming up with scheming strategies (which is probably what I care about most).

  I’d probably rather think about it in terms of, “Tasks that previously required at least N tokens of CoT can now be done without CoT,” rather than this roundabout argument with number of layers needed to double no-CoT time horizons.

  I think you’re saying that getting an 8.2x increase in effective depth equals 1.3^8, and each 1.3x in layers yields a doubling in no-CoT time horizon, which covers the 2^8 gap that exists right now between no-CoT and unlimited CoT time horizons.

  This yields a conflict between the two interpretations. I’d probably want to interpret 8 doublings in no-CoT serial reasoning ability as enabling hidden reasoning equivalent to using 2^8=256 tokens in current models. Which is probably enough to do lots of scheming reasoning, but not equivalent to unlimited CoT (which is allowed for normal time horizons). I’m not exactly sure how to reconcile these views. Like I said, I find the “Tasks that previously required at least N tokens of CoT can now be done without CoT” framing more intuitive and compelling, so by default I’d take that perspective.

• RohanS 27 Aug 2026 17:55 UTC

  5 points

  0

  in reply to: Rauno Arike’s comment on: Brief in­de­pen­dent in­ves­ti­ga­tion of agents’ be­hav­ior, rea­son­ing and col­lab­o­ra­tion in the OpenAI /​ Hug­ging Face hack­ing incident

  an alert saying that the agent is trying to make an account on Hugging Face or trying to download a private dataset won’t be that surprising on its own for that employee.

  I’m sympathetic to the general concern that it could be hard to surface the truly egregious misbehaviors given the false positive rates we may see. But in this case, the alert should say the agent was trying to cheat on the task, coordinate with other cheating agents on a message board that wasn’t intended to exist, and hack Hugging Face!

  That’s what knowledgeable humans would have to say if they read any one agent’s transcript (I think). And if one knowledgeable human was watching another human while they did all of this stuff, after only having been asked to solve the ExploitBench problem, I think it would be easy to flag it as extremely bad while still having a low false positive rate for desirable behaviors.

  I think one problem here is that LLMs have to infer a lot about their contexts in ways that humans don’t. I suppose they might think this is how this ExploitBench task was supposed to go; that it was an impossible challenge and the message board was intentionally provided, or something like that. But if you just specifically prompt an LLM monitor to look for egregious misbehaviors relative to what the human wanted, I would expect them to flag this. (Seems like a potentially good use case for extract-and-evaluate monitoring, actually :).)

  OpenAI did say in their report, “We specifically improved our monitor’s recall over the types of behavior that occurred in the incident using the hindsight of our investigation,” which I think is straightforwardly good, but it’s hard to know how robust the monitors are going to be.

• RohanS 26 Aug 2026 23:09 UTC

  10 points

  −2

  in reply to: draganover’s comment on: Brief in­de­pen­dent in­ves­ti­ga­tion of agents’ be­hav­ior, rea­son­ing and col­lab­o­ra­tion in the OpenAI /​ Hug­ging Face hack­ing incident

  What aspect of this report makes you update that way? Or did you already think this?

  I think the scale of agent reasoning and interactions is in line with what I expected, and I just want AI companies to monitor way more of their agents’ CoTs. (I broadly liked the “Expanding chain-of-thought monitoring” section of OpenAI’s temporary slowdown post.)

  And wrt “impenetrability of the model outputs,” some of the reasoning traces in the figures reminded me that these are getting less human-legible; but for now they still seem reasonably human-legible, and I think maybe companies should make some interventions to make them more legible. This might include explicitly training for CoT legibility.

  Of course, there’s a well-known objection that I expect many people to have: “This is the Most Forbidden Technique, it’s a terrible idea!” I’ve thought and written about this; I think there are a ton of different types of training against CoT, and the risk of getting obfuscated reasoning varies a lot between them. The most classic bad form of this training is to penalize reward hacks while also rewarding task completion, which makes the optimal behavior obfuscated reward hacks. But if you have a phase of training that only rewards legible reasoning, there’s no conflict. And maybe if all you do is paraphrase slightly hard-to-read traces before reinforcing them, you never get far enough away from the legible starting point to incentivize obfuscation. (I’d have to think a bit more about the details of implementing that during GRPO to be confident, and if companies are doing something better than GRPO there are probably other considerations there.) This post, and especially the Interlude section, are where I’ve written about this before.

  I still think that switching to recurrent neuralese, linguistic drift, steganography, not monitoring enough trajectories, and more single-forward-pass serial reasoning ability are all plausible ways CoT monitoring could break down, but “within a year, CoT monitoring will no longer be a sensible solution” seems too strong to me.

  (Maybe I should look more closely at No-CoT time horizons, my guess is that those are growing but we don’t seem close to AIs that can come up with nontrivial scheming strategies without CoT.)

• RohanS 19 Aug 2026 5:50 UTC

  25 points

  0

  on: Ro­hanS’s Shortform

  Between Ryan Greenblatt’s Current AIs seem pretty misaligned to me and Steven Byrnes’s Four LLM loss functions → four flavors of LLM misalignment, I think my overall view on the state of alignment has shifted somewhat more pessimistically.

  My old view: The models seem pretty aligned right now.

  My current view (open to feedback):

  —

  ‘Aligned behavior’ is a very complicated thing and a pretty narrow target, even when interpreted as ‘roughly instruction-following behavior without any egregious misaligned behaviors.’ There are a ton of ways to be misaligned. Every training signal we’ve tried on LLMs causes some kind of misalignment, by which I mean that there are important divergences between model behavior and developer intentions. [See the table at the top of the Steven Byrnes post.] It seems like reward hacking is way harder to stop than I had expected, and I think I understand why now: making AIs persistent and creative enough to solve hard tasks is in tension with making AIs cautious about how they go about doing tasks. It still feels like LLMs should have learned a lot of common sense from pretraining, and there was some chance that it would be easy for them to at least avoid egregious reward hacks like in the recent OpenAI Hugging Face and UK AISI incidents, but it seems like that boundary is in fact somewhat complicated so AI companies have failed to instill it into their models effectively.

  This makes the analogy to humans feel a bit more real. (I’ve written about this before here.) We use a ton of proxies to teach humans how to behave well; ~all of them can be Goodharted; ~everyone succeeds according to some proxies and fails according to others; sometimes (often?) people live bad lives; and it’s especially likely that people live bad lives according to big picture criteria that only have slow or nonexistent feedback loops (aka, that are hard-to-verify) and that conflict with proxies with faster feedback loops. E.g., ‘the net effect of your life on the well-being of factory farmed animals’ has an almost nonexistent feedback loop for most people, while ‘the taste of meat’ has a fast feedback loop. (I might want to make a table like the one at the top of Steven Byrnes’s post, but for human misalignment instead of LLM misalignment. I’d start by looking at Section 9 of my post here (also linked above). I’d love it if someone else did this first!)

  In other words, with humans it feels like a bit of a whack-a-mole of different proxies, and after that it’s a toss-up whether we get alignment or not. And I’m starting to think the situation is kind of similar in LLMs.

  There are at least a couple important differences though:

  1) Humans seem to have better alignment generalization properties (for now), e.g. it’s rare to see anything as bad as the OpenAI Huggingface incident from typical humans, it’s not that hard to teach humans that persistence vs non-reward-hacking boundary. This feels like it might be a valuable target for our alignment work imo: make it so that with the same set of proxies, AIs are no more misaligned than humans. (Probably Steven Byrnes has written about this a bunch too, I’d like to look into that more; something about our innate reward functions and how we update our beliefs separately from our desires :).)

  2) AIs are more monitorable for now in principle; I think if we read all CoTs, we’re getting much more oversight than human parents get over their kids (who get to think long serial thoughts to themselves and take lots of actions while out of sight).

  (Side note: Frankly, I’m highly unsure if I’m acting aligned relative to my most important goals like ‘make the long-term future as good as possible,’ because the feedback loop is nonexistent and the proxies I use for selecting actions (like ‘what I think seems good based on my current AI safety takes’) are highly imperfect.)

  —

  Even before these posts and incidents, I already thought there could be big problems that arise as AIs become more capable of much longer-horizon tasks, because the capacity to do those tasks inherently requires models to make cognitive moves that could break alignment, like reflecting on how to pursue their goals and abandoning subgoals. CoT seems monitorable enough that I think we’d be able to notice all those problems if we stay close to the current regime; I get a lot more concerned if we switch to recurrent neuralese.

  (Written quickly, maybe some things I won’t endorse on reflection)

  Edit: I want a shorter handle for my new view, so I don’t accidentally revert to the ‘things are fine’ view just because it’s easier to remember in contexts where I don’t have much time to think about it. I think I’ll call my new view on alignment the “whack-a-mole + toss-up” view, at least in my own head. And if I want to recall some of the most important pieces that are missing from that, a slightly longer handle might be “proxy whack-a-mole + toss-up + good monitoring + long-horizon cognitive moves.”

• RohanS 13 Aug 2026 3:22 UTC

  8 points

  1

  in reply to: leogao’s comment on: leogao’s Shortform

  Even if there are better things to do in the future, it seems like one of the better things to do now! This feels like a very general argument against doing almost anything now. Maybe it doesn’t apply to some types of alignment work that interact little with any specific details of current models and take a lot of serial time to get working. I’m curious if you think there are broad swaths of things that might all be better to work on because of the argument you gave.

  It also probably depends what you consider to be ‘studying the HF incident.’ Eg, studying ‘why RL sometimes results in models that desperately want to succeed at tasks with little regard for side effects’ seems very important, and the HF incident seems like a great example to center that study around, but that work may not involve spending too much time looking at details of the incident.

• RohanS 7 Aug 2026 7:18 UTC

  3 points

  0

  on: Ro­hanS’s Shortform

  I’ve gotten good mileage out of buying lots of socks and underwear. They’re always my laundry bottleneck, and they’re quite cheap. My laundry cycle is now >2 weeks. Maybe you should buy/​order more socks or underwear.

Per­spec­tives on Con­tinual Learn­ing: Sur­vey Re­sults and Forecasts

An­gles of at­tack for con­tinual learn­ing safety

How might con­tinual learn­ing af­fect safety and al­ign­ment?

• RohanS 12 Jun 2026 21:22 UTC

  2 points

  1

  in reply to: RogerDearnaley’s comment on: What’s Con­tinual Learn­ing, and Why Might We Ex­pect To See It In Ad­vanced LLM Agents?

  We’re generally intelligent, but we’d be even more effective continual learners if we did less (catastrophic) forgetting! I think this essay may be on the right track with pointing to ‘general capabilities’ as the thing you don’t want to forget: Defining Continual Learning

• RohanS 12 Jun 2026 19:10 UTC

  3 points

  −2

  on: PSA: Al­most no­body is di­rectly work­ing on su­per­in­tel­li­gent alignment

  I agree that a small fraction of the AI safety field is working on any version of superintelligence alignment.

  I think alignment training (character training /​ constitutional AI /​ deliberative alignment /​ RLHF) is a major bucket of work on alignment, and I think it’s very reasonable to see it as part of an iterative strategy for superintelligence alignment. (You kind of gesture towards this, but you don’t mention alignment training explicitly.)

  I recently discovered this post from 2023: Ten Levels of AI Alignment Difficulty. I think it does a good job showing how very different types of work get called ‘alignment research,’ and how they’re conditioned on different views about what needs to happen to build aligned powerful AIs. It includes simple RLHF-style alignment training, scalable oversight, and agent foundations.

  Shameless plug: I recently wrote a post that’s mostly about the pros and cons of various forms of alignment training, but also touches on alternatives to alignment training, which contains most of my up-to-date thoughts about alignment. It’s called Should We Train Against (CoT) Monitors?

What’s Con­tinual Learn­ing, and Why Might We Ex­pect To See It In Ad­vanced LLM Agents?

Im­pli­ca­tions of Con­tinual Learn­ing for LLM Agents: Introduction

• RohanS 17 May 2026 6:31 UTC

  1 point

  0

  in reply to: Daniel Tan’s comment on: Ro­hanS’s Shortform

  Sorry I only skimmed your old post, but I’m confused how something like truesight could enable general-purpose hidden reasoning that looks normal? I agree that hidden reasoning is the concern and the exact mechanism doesn’t matter, and your version of steg seems slightly more plausible, but I’d still say steg is less likely than other failure modes.

  So IDK, I feel like people should update all the way towards ‘unmonitored information transmission is a propensity thing not a capability thing’

  I don’t buy this, I think models are totally incapable of some important kinds of hidden serial reasoning, eg figuring out how to self-exfiltrate. (And we shouldn’t write about this online, because that could make it way easier.) Also, I think if you’ve been trained to have a propensity to verbalize things, that probably makes it much more difficult to not verbalize them even if you want to, so propensity and capability are far from independent here.

• RohanS 16 May 2026 16:30 UTC

  1 point

  0

  in reply to: papetoast’s comment on: Ro­hanS’s Shortform

  Because taking over the world or getting to a position where CoT is fully unmonitored seems pretty hard for an LLM! As long as we don’t write too much about specific strategies for how to do these things online, they’d probably have to come up with sophisticated strategies on their own, which I think likely requires serial reasoning. (On the other hand they may just need to keep an eye out for easy opportunities for self-exfiltration that don’t require serial reasoning; it’s unclear to me if such opportunities will ever present themselves.)

• RohanS 16 May 2026 6:47 UTC

  2 points

  0

  in reply to: kbear’s comment on: Ro­hanS’s Shortform

  Well played :)

  I’m not sure I buy this though:

  so it may hide some idea, when it does what it does. and what is more, it won’t need to say how to hide the idea out loud. no—by the way it is made it will pick up on any such idea if you just give it a few lead-ins. its main jam is to find and add to this sort of game.

  In particular, I’m confused how something like this could allow for hiding arbitrary reasoning /​ messages without looking suspicious

  to pick its next word with care—at this it will be deft.

  To some extent I buy this, I can imagine LLMs can choose to write an essay with only words less than 5 letters. (Opus 4.7 mostly succeeded in one attempt I just gave it.) But also, this isn’t sufficient to achieve general-purpose hidden serial reasoning. It’s much harder to pick tokens based on synonym partitions (or number-of-letter partitions) in order to communicate a specific hidden message or do reasoning than to just always use words of at most 4 letters with no hidden message or reasoning.

• RohanS 16 May 2026 0:37 UTC

  20 points

  −7

  on: Ro­hanS’s Shortform

  I’ve felt for a little while now that steganography, in the sense of text that looks normal and monitorable to humans but actually allows LLMs to do general-purpose hidden reasoning, is a less likely failure mode of CoT monitoring than switching to recurrent latent reasoning (neuralese) or obvious and visible linguistic drift.

  I think I have a new and fairly crisp argument for this: Using general-purpose steganographic encoding and decoding itself requires a lot of serial reasoning.

  For example, one way you can do steganography is by partitioning the set of all tokens into disjoint groups, with synonyms placed in different sets, and every time you have multiple words to choose from with the same meaning, you encode some additional hidden meaning based on which set the word you output belongs to. To reliably output the right token to encode meaning by such a scheme, you probably need to think pretty carefully about how to follow some rules about which tokens are in which set and what you are trying to encode. Similarly, extracting the hidden message from the text requires carefully pulling out the hidden bits and then piecing together what they mean. So either the serial reasoning required for encoding and decoding should be easily monitorable, or there is some non-steganographic reason why that serial reasoning is unmonitorable. Either way, steganography isn’t the first reason why CoT monitoring fails.

  I would guess this is true of all general-purpose steganographic encoding and decoding schemes because it seems hard to hide arbitrary messages in normal-looking text without thinking hard about what to write, but I’m not super confident about that.