RohanS

• RohanS 12 Jun 2026 21:22 UTC

  2 points

  1

  in reply to: RogerDearnaley’s comment on: What’s Con­tinual Learn­ing, and Why Might We Ex­pect To See It In Ad­vanced LLM Agents?

  We’re generally intelligent, but we’d be even more effective continual learners if we did less (catastrophic) forgetting! I think this essay may be on the right track with pointing to ‘general capabilities’ as the thing you don’t want to forget: Defining Continual Learning

• RohanS 12 Jun 2026 19:10 UTC

  3 points

  −2

  on: PSA: Al­most no­body is di­rectly work­ing on su­per­in­tel­li­gent alignment

  I agree that a small fraction of the AI safety field is working on any version of superintelligence alignment.

  I think alignment training (character training /​ constitutional AI /​ deliberative alignment /​ RLHF) is a major bucket of work on alignment, and I think it’s very reasonable to see it as part of an iterative strategy for superintelligence alignment. (You kind of gesture towards this, but you don’t mention alignment training explicitly.)

  I recently discovered this post from 2023: Ten Levels of AI Alignment Difficulty. I think it does a good job showing how very different types of work get called ‘alignment research,’ and how they’re conditioned on different views about what needs to happen to build aligned powerful AIs. It includes simple RLHF-style alignment training, scalable oversight, and agent foundations.

  Shameless plug: I recently wrote a post that’s mostly about the pros and cons of various forms of alignment training, but also touches on alternatives to alignment training, which contains most of my up-to-date thoughts about alignment. It’s called Should We Train Against (CoT) Monitors?

What’s Con­tinual Learn­ing, and Why Might We Ex­pect To See It In Ad­vanced LLM Agents?

Im­pli­ca­tions of Con­tinual Learn­ing for LLM Agents: Introduction

• RohanS 17 May 2026 6:31 UTC

  1 point

  0

  in reply to: Daniel Tan’s comment on: Ro­hanS’s Shortform

  Sorry I only skimmed your old post, but I’m confused how something like truesight could enable general-purpose hidden reasoning that looks normal? I agree that hidden reasoning is the concern and the exact mechanism doesn’t matter, and your version of steg seems slightly more plausible, but I’d still say steg is less likely than other failure modes.

  So IDK, I feel like people should update all the way towards ‘unmonitored information transmission is a propensity thing not a capability thing’

  I don’t buy this, I think models are totally incapable of some important kinds of hidden serial reasoning, eg figuring out how to self-exfiltrate. (And we shouldn’t write about this online, because that could make it way easier.) Also, I think if you’ve been trained to have a propensity to verbalize things, that probably makes it much more difficult to not verbalize them even if you want to, so propensity and capability are far from independent here.

• RohanS 16 May 2026 16:30 UTC

  1 point

  0

  in reply to: papetoast’s comment on: Ro­hanS’s Shortform

  Because taking over the world or getting to a position where CoT is fully unmonitored seems pretty hard for an LLM! As long as we don’t write too much about specific strategies for how to do these things online, they’d probably have to come up with sophisticated strategies on their own, which I think likely requires serial reasoning. (On the other hand they may just need to keep an eye out for easy opportunities for self-exfiltration that don’t require serial reasoning; it’s unclear to me if such opportunities will ever present themselves.)

• RohanS 16 May 2026 6:47 UTC

  2 points

  0

  in reply to: kbear’s comment on: Ro­hanS’s Shortform

  Well played :)

  I’m not sure I buy this though:

  so it may hide some idea, when it does what it does. and what is more, it won’t need to say how to hide the idea out loud. no—by the way it is made it will pick up on any such idea if you just give it a few lead-ins. its main jam is to find and add to this sort of game.

  In particular, I’m confused how something like this could allow for hiding arbitrary reasoning /​ messages without looking suspicious

  to pick its next word with care—at this it will be deft.

  To some extent I buy this, I can imagine LLMs can choose to write an essay with only words less than 5 letters. (Opus 4.7 mostly succeeded in one attempt I just gave it.) But also, this isn’t sufficient to achieve general-purpose hidden serial reasoning. It’s much harder to pick tokens based on synonym partitions (or number-of-letter partitions) in order to communicate a specific hidden message or do reasoning than to just always use words of at most 4 letters with no hidden message or reasoning.

• RohanS 16 May 2026 0:37 UTC

  20 points

  −7

  on: Ro­hanS’s Shortform

  I’ve felt for a little while now that steganography, in the sense of text that looks normal and monitorable to humans but actually allows LLMs to do general-purpose hidden reasoning, is a less likely failure mode of CoT monitoring than switching to recurrent latent reasoning (neuralese) or obvious and visible linguistic drift.

  I think I have a new and fairly crisp argument for this: Using general-purpose steganographic encoding and decoding itself requires a lot of serial reasoning.

  For example, one way you can do steganography is by partitioning the set of all tokens into disjoint groups, with synonyms placed in different sets, and every time you have multiple words to choose from with the same meaning, you encode some additional hidden meaning based on which set the word you output belongs to. To reliably output the right token to encode meaning by such a scheme, you probably need to think pretty carefully about how to follow some rules about which tokens are in which set and what you are trying to encode. Similarly, extracting the hidden message from the text requires carefully pulling out the hidden bits and then piecing together what they mean. So either the serial reasoning required for encoding and decoding should be easily monitorable, or there is some non-steganographic reason why that serial reasoning is unmonitorable. Either way, steganography isn’t the first reason why CoT monitoring fails.

  I would guess this is true of all general-purpose steganographic encoding and decoding schemes because it seems hard to hide arbitrary messages in normal-looking text without thinking hard about what to write, but I’m not super confident about that.

• RohanS 24 Apr 2026 7:14 UTC

  2 points

  0

  in reply to: Bronson Schoen’s comment on: Should We Train Against (CoT) Mon­i­tors?

  First of all, I think a lot of what you’ve said here is reasonable and I appreciate the thoughtful responses!

  Still, I’m overall less pessimistic than you.

  I think there are other types of evidence and argument we can use for deciding whether our training methods are likely to select for obfuscated misbehavior or aligned behavior. For instance, we can just… think through the training setup and intuitively see how likely we think each outcome is. Of course, this is not a rigorous method, and people are sure to have different intuitions, and in general, intuitions about the outcomes of ML/​DL training runs aren’t good. But this is still a good starting point for saying things like “it’s worse to penalize alignment faking in the CoT every time you see it than to do deliberative alignment with a non-alignment-faking safety spec.” Then we can try to be more specific and careful about what aspects of the training setup underlie this intuition.

  I also think that it’s not super clear that having held-out ways of measuring whether a model is aligned is as essential to aligning models as having a held-out set in typical ML situations. It feels more like an analogy than like the same thing, and there are other analogies that point the other way and seem reasonable too. One such analogy is that parents don’t often avoid teaching their children how to behave well in a bunch of situations just to be able to evaluate whether their other efforts to raise the kid right are generalizing well. Ok honestly maybe parents sometimes do this, and maybe parents should do this more. It’s not a clean argument in favor of my point. But I still think it’s probably possible to come up with some analogies that suggest we should do this kind of training.

  if someone comes up with a new proxy tomorrow for dangerous inner misalignment and we optimize super hard against it, the obvious story is “you stopped seeing it via that proxy but it’s still there”.

  Yeah, agreed. I was being too vague in my demand for specific stories. I think the people training with proxies should make sure their setups don’t fall prey to the well-known pitfalls, and they should take into account arguments of any kind that their setup could result in obfuscated misbehavior. But for some setups (especially those described in Sections 4 and 5), I think there may be no strong default of obfuscation, nor any clear story for why it would be dangerous, which I think makes it at least worth studying. And I still agree that it would be good to have held-out signals, but I think it might be worthwhile to force ourselves to use eval signals that are more correlated with training signals if that lets us do much more to train out bad behavior with methods that have a realistic chance of generalizing.

  Re: Deliberative Alignment, I made my own argument for why I think it’s probably fine (and you quoted it) rather than deferring to OpenAI. (In fact, I’ve heard OpenAI people say they’re concerned about obfuscation from Deliberative Alignment too.) Your response to Rauno where you said pre-safety o3 seemed more monitorable was interesting and could be an update for me here, but it’s also quite surprising and I’d be curious how much evidence you have for DA being responsible for that. I’d also be curious how monitorable more recent OpenAI CoT’s are, eg for GPT-5.5. I’ve heard third-hand that recent models are more legible than o3, it would be great if OpenAI could share a small number of transcripts either CoT’s to make this more widely known.

• RohanS 23 Apr 2026 22:27 UTC

  3 points

  0

  in reply to: Daniel Kokotajlo’s comment on: Should We Train Against (CoT) Mon­i­tors?

  I partially agree: I think held-out tests are a great way to verify aligned cognition, especially if we never train on anything close to making that cognition look aligned. But also, it might be hard to get aligned cognition in high-stakes deployments if we never train on anything close to that. There are some unresolved tensions here within my post: Section 3 aligns with your point, while Sections 4 and 5 offer my best current ideas for getting the benefits of training without losing monitorability. (You can just read the section titles for the gist.)

  If we use targeted interventions as suggested in Section 5, especially when using multiple strong and diverse proxies as suggested in Section 4, I think we may be able to build some confidence that a) it is implausible for this training setup to teach obfuscated reasoning and b) this training setup will teach aligned reasoning/​behavior/​motivations that generalize at least moderately well. I think there’s quite a bit of room for advancing our understanding of how to do this well (and whether it’s possible), especially with experiments but maybe also with more conceptual arguments.

Should We Train Against (CoT) Mon­i­tors?

• RohanS 15 Apr 2026 2:49 UTC

  9 points

  3

  in reply to: Shmi’s comment on: Morale

  Morale comes from having the nice things in your life correlated with effort.

  Hmm I think this is only sort of true, and my disagreement stems from my disagreement with the provided definition of morale.

  Morale is, roughly, “the belief that if you work hard, your conditions will improve.”

  My first pass rephrasal is:

  Morale is, roughly, “the belief that if you try, your conditions will improve.”

  I want to capture that sometimes you want to work smarter, not harder, and indeed, getting great outcomes for low-effort-but-smart solutions is extremely satisfying and seems like a central type of morale-building experience imo. Admittedly you can interpret “effort” and “work hard” in ways that automatically capture this, but I feel like this point was missing.

  (This is more a response to the post than your comment, I like the connection to learned helplessness.)

• RohanS 3 Apr 2026 20:34 UTC

  5 points

  0

  on: There should be $100M grants to au­to­mate AI safety

  I think my main concern is the Goodhart one.

  For the proxy you gave, is the idea that if you can detect a huge number of “distinct, egregiously misaligned ‘features’ found by interpretability (and a human expert would judge them as egregious, diverse and high quality),” then labs could use your methods and have a high likelihood of detecting egregious misalignment if it exists? $100M is a nontrivial fraction of all the funding available for AI x-risk reduction, so in some vague sense I feel like doing very well according to the proxies needs to mean solving a nontrivial fraction of all the x-risk from AI in order for these grants to make sense. Maybe this does meet the bar, I’d have to think more.

  (I’m a bit unsure what it even means for a feature to be egregiously misaligned, won’t even aligned models recognize egregiously bad behaviors and therefore have features for them?)

• RohanS 24 Feb 2026 17:59 UTC

  LW: 4 AF: 4

  0

  AF

  in reply to: RohanS’s comment on: Why we should ex­pect ruth­less so­ciopath ASI

  [Edit: I don’t think this is saying anything that different than my comment above, but it is a slightly different framing.]

  Another point that I think might be quite important: we often set ourselves complex subgoals in line with our existing values, and then we try hard to achieve those goals, and we learn how to be more effective consequentialist agents at achieving that type of subgoal. There may be clearer feedback on how well we did at the subgoal than how well we achieved our existing values, but in lots of cases we notice if there’s a significant divergence between what we achieved and our underlying values, which moderates the consequentialist learning and is a pressure towards maintaining alignment.

• RohanS 24 Feb 2026 6:47 UTC

  LW: 8 AF: 5

  0

  AF

  in reply to: Steven Byrnes’s comment on: Why we should ex­pect ruth­less so­ciopath ASI

  (Slightly rambly comment, sorry)

  • I agree open-ended continual learning (CL) is probably big, I have been thinking and writing about CL a bunch recently but tbh I don’t think I’m near the end of clarifying all my thoughts on it. (Still hope to publish a sequence on it with some collaborators soon though.)

  I think it’s well known that LLMs struggle with large amounts of interrelated complexity in the training data, when none of that complexity, or anything remotely related to it, was in the pretraining data. “Open-ended continual learning purely through the context window” would be a very extreme version of that—centuries of knowledge, entire new fields and ways of thinking, entirely absent from pretraining and exclusively in the context window. No way is that going to work.

  • I agree weight updates are probably needed, I like the way you phrased the limitation, it matches some thoughts I’ve had but never as precisely.

  • I expect you understand continual learning and especially the brain better than I do, but it seems plausible to me that your interpretation of the alignment implications on top of that understanding is flawed.

  • I think “asymptotically 100% consequentialist” is quite possibly wrong about the objectives used for open-ended CL training.

    • We can incorporate ongoing character training to ensure it has non-negligible asymptotic representation.

      • I think humans interpret a lot of experiences we have in the context of our existing values, and this informs how we update. This can frequently reinforce our values.

      • Self-verification seems like it may be an important part of the CL objectives, and this can include self-verification of alignment with existing character (which starts close to Claude’s current nice character and hopefully stays close with some desirable ironing-out).

      • Maybe this “doing continual learning informed by existing values” is kind of similar to humans doing continual learning informed by human social instincts? I also think this is related to the confusion that I and some other commenters have about why imitation and consequentialism are the only options. I have a much messier list of possible update mechanisms that doesn’t seem like it fits cleanly into those two as broad categories. Maybe a good example is that humans update on a ton of random observations we’re surprised by. This doesn’t seem like imitation, nor does it seem consequentialist enough to be very risky? (Maybe there’s an active inference-related case to be made for consequentialism here, but I haven’t looked into that much, I’d be curious for someone to make that argument if so.)

      • “Increasing quantity and quality of character training throughout continual learning” seems like a potentially promising avenue for interventions, do you agree?

  What links here?

  • Should We Train Against (CoT) Mon­i­tors? by RohanS (23 Apr 2026 19:19 UTC; 50 points)

• RohanS 23 Feb 2026 17:46 UTC

  8 points

  −2

  on: Ro­hanS’s Shortform

  I think quite a lot of the disagreement about AGI timelines comes from differing attitudes towards benchmarks. It’s unclear to me how big the gap is between benchmarks and real-world applications. Benchmarks are increasingly trying to evaluate more real-world tasks and incorporate subjective evaluation where it’s difficult to do purely automated evaluation. I think I’m now of the opinion that if you ignore benchmarks and think that improvement on benchmarks isn’t much indication of progress towards truly transformative AI, you’re making a mistake. (But I’m not sure.)

  One reason for this is that if you start with that attitude, then you can think about what would make for better benchmarks that actually measure the thing you care about. And it turns out a bunch of other people have done that and have been making better and better benchmarks, and there’s progress on most of those. And the things that I think really matter for AGI capabilities, like automating AI research, seem to be progressing. I have remaining uncertainty about whether even these new benchmarks and evaluations are catering to current capabilities too much, it’s not impossible to me that there are big missing pieces.

• RohanS 22 Feb 2026 20:05 UTC

  LW: 5 AF: 4

  0

  AF

  in reply to: Steven Byrnes’s comment on: Why we should ex­pect ruth­less so­ciopath ASI

  I’d be curious for you to say a bit more in response to this point from above:

  It seems like you are impliying LLM have something like the human social insticts via imitation already at inference time but you can’t use them in any way to boostrap to some continuous learning thing thats grounded in human-like consequentialism and that seems like thats the direction were interesting discusion lies ?.

  I’m moderately optimistic about our ability to get roughly human-like consequentialism from LLM-based AGI, with character training instilling a non-ruthless non-sociopath character that is still compatible with lots of consequentialist agency, like very good scientists/​inventors/​entrepreneurs/​etc. who never do anything that could be very dangerous (because a bunch of useful things aren’t that risky, and because they have good enough moral principles to deliberately avoid or be careful around things that are risky).

  I think long-horizon RL or reflection or other components of the continual learning process could break the instilled character, but it seems >50% likely to me that between the preliminary character training and ongoing training and prompting to maintain good character, those things won’t dominate, and we’ll just have nice AIs. (I get a bit more nervous about this argument for ASI, but I think it may well hold up even there.)

[Paper] How does in­for­ma­tion ac­cess af­fect LLM mon­i­tors’ abil­ity to de­tect sab­o­tage?

• RohanS 3 Feb 2026 12:57 UTC

  2 points

  0

  on: Ro­hanS’s Shortform

  Just went on my first podcast! It was a pleasure to discuss continual learning for LLM agents and its safety implications with Anna on The Glitchatorio, you can check it out at either of the below links:

  Spotify: https://​​open.spotify.com/​​episode/​​4cU4aSMSgmstzPJ9ijglvc?si=DLPPfXbKSk233azzIspSkQ&t=218

  Apple Podcasts: https://​​podcasts.apple.com/​​de/​​podcast/​​the-glitchatorio/​​id1836777868?l=en-GB&i=1000747658363

• RohanS 23 Jan 2026 17:30 UTC

  1 point

  0

  in reply to: avturchin’s comment on: A quick, el­e­gant deriva­tion of Bayes’ Theorem

  I think the example has more to do with applying the theorem, while the post is about deriving it, so I’m not sure the post helps with the example directly.