RohanS

• RohanS 24 Feb 2026 17:59 UTC

  LW: 4 AF: 4

  0

  AF

  in reply to: RohanS’s comment on: Why we should ex­pect ruth­less so­ciopath ASI

  [Edit: I don’t think this is saying anything that different than my comment above, but it is a slightly different framing.]

  Another point that I think might be quite important: we often set ourselves complex subgoals in line with our existing values, and then we try hard to achieve those goals, and we learn how to be more effective consequentialist agents at achieving that type of subgoal. There may be clearer feedback on how well we did at the subgoal than how well we achieved our existing values, but in lots of cases we notice if there’s a significant divergence between what we achieved and our underlying values, which moderates the consequentialist learning and is a pressure towards maintaining alignment.

• RohanS 24 Feb 2026 6:47 UTC

  LW: 8 AF: 5

  0

  AF

  in reply to: Steven Byrnes’s comment on: Why we should ex­pect ruth­less so­ciopath ASI

  (Slightly rambly comment, sorry)

  • I agree open-ended continual learning (CL) is probably big, I have been thinking and writing about CL a bunch recently but tbh I don’t think I’m near the end of clarifying all my thoughts on it. (Still hope to publish a sequence on it with some collaborators soon though.)

  I think it’s well known that LLMs struggle with large amounts of interrelated complexity in the training data, when none of that complexity, or anything remotely related to it, was in the pretraining data. “Open-ended continual learning purely through the context window” would be a very extreme version of that—centuries of knowledge, entire new fields and ways of thinking, entirely absent from pretraining and exclusively in the context window. No way is that going to work.

  • I agree weight updates are probably needed, I like the way you phrased the limitation, it matches some thoughts I’ve had but never as precisely.

  • I expect you understand continual learning and especially the brain better than I do, but it seems plausible to me that your interpretation of the alignment implications on top of that understanding is flawed.

  • I think “asymptotically 100% consequentialist” is quite possibly wrong about the objectives used for open-ended CL training.

    • We can incorporate ongoing character training to ensure it has non-negligible asymptotic representation.

      • I think humans interpret a lot of experiences we have in the context of our existing values, and this informs how we update. This can frequently reinforce our values.

      • Self-verification seems like it may be an important part of the CL objectives, and this can include self-verification of alignment with existing character (which starts close to Claude’s current nice character and hopefully stays close with some desirable ironing-out).

      • Maybe this “doing continual learning informed by existing values” is kind of similar to humans doing continual learning informed by human social instincts? I also think this is related to the confusion that I and some other commenters have about why imitation and consequentialism are the only options. I have a much messier list of possible update mechanisms that doesn’t seem like it fits cleanly into those two as broad categories. Maybe a good example is that humans update on a ton of random observations we’re surprised by. This doesn’t seem like imitation, nor does it seem consequentialist enough to be very risky? (Maybe there’s an active inference-related case to be made for consequentialism here, but I haven’t looked into that much, I’d be curious for someone to make that argument if so.)

      • “Increasing quantity and quality of character training throughout continual learning” seems like a potentially promising avenue for interventions, do you agree?

• RohanS 23 Feb 2026 17:46 UTC

  8 points

  −2

  on: Ro­hanS’s Shortform

  I think quite a lot of the disagreement about AGI timelines comes from differing attitudes towards benchmarks. It’s unclear to me how big the gap is between benchmarks and real-world applications. Benchmarks are increasingly trying to evaluate more real-world tasks and incorporate subjective evaluation where it’s difficult to do purely automated evaluation. I think I’m now of the opinion that if you ignore benchmarks and think that improvement on benchmarks isn’t much indication of progress towards truly transformative AI, you’re making a mistake. (But I’m not sure.)

  One reason for this is that if you start with that attitude, then you can think about what would make for better benchmarks that actually measure the thing you care about. And it turns out a bunch of other people have done that and have been making better and better benchmarks, and there’s progress on most of those. And the things that I think really matter for AGI capabilities, like automating AI research, seem to be progressing. I have remaining uncertainty about whether even these new benchmarks and evaluations are catering to current capabilities too much, it’s not impossible to me that there are big missing pieces.

• RohanS 22 Feb 2026 20:05 UTC

  LW: 5 AF: 4

  0

  AF

  in reply to: Steven Byrnes’s comment on: Why we should ex­pect ruth­less so­ciopath ASI

  I’d be curious for you to say a bit more in response to this point from above:

  It seems like you are impliying LLM have something like the human social insticts via imitation already at inference time but you can’t use them in any way to boostrap to some continuous learning thing thats grounded in human-like consequentialism and that seems like thats the direction were interesting discusion lies ?.

  I’m moderately optimistic about our ability to get roughly human-like consequentialism from LLM-based AGI, with character training instilling a non-ruthless non-sociopath character that is still compatible with lots of consequentialist agency, like very good scientists/​inventors/​entrepreneurs/​etc. who never do anything that could be very dangerous (because a bunch of useful things aren’t that risky, and because they have good enough moral principles to deliberately avoid or be careful around things that are risky).

  I think long-horizon RL or reflection or other components of the continual learning process could break the instilled character, but it seems >50% likely to me that between the preliminary character training and ongoing training and prompting to maintain good character, those things won’t dominate, and we’ll just have nice AIs. (I get a bit more nervous about this argument for ASI, but I think it may well hold up even there.)

[Paper] How does in­for­ma­tion ac­cess af­fect LLM mon­i­tors’ abil­ity to de­tect sab­o­tage?

• RohanS 3 Feb 2026 12:57 UTC

  2 points

  0

  on: Ro­hanS’s Shortform

  Just went on my first podcast! It was a pleasure to discuss continual learning for LLM agents and its safety implications with Anna on The Glitchatorio, you can check it out at either of the below links:

  Spotify: https://​​open.spotify.com/​​episode/​​4cU4aSMSgmstzPJ9ijglvc?si=DLPPfXbKSk233azzIspSkQ&t=218

  Apple Podcasts: https://​​podcasts.apple.com/​​de/​​podcast/​​the-glitchatorio/​​id1836777868?l=en-GB&i=1000747658363

• RohanS 23 Jan 2026 17:30 UTC

  1 point

  0

  in reply to: avturchin’s comment on: A quick, el­e­gant deriva­tion of Bayes’ Theorem

  I think the example has more to do with applying the theorem, while the post is about deriving it, so I’m not sure the post helps with the example directly.

• RohanS 23 Jan 2026 17:27 UTC

  5 points

  2

  in reply to: Josh Snider’s comment on: A quick, el­e­gant deriva­tion of Bayes’ Theorem

  That seems philosophically deeper than the goal of this post, which is to just help people who already accept the standard joint probability statement to remember Bayes’ Theorem. If I want to remember or write Bayes’ Theorem, I roughly just think “joint probability, divide.”

A quick, el­e­gant deriva­tion of Bayes’ Theorem

Ex­plor­ing Re­in­force­ment Learn­ing Effects on Chain-of-Thought Legibility

Aether is hiring tech­ni­cal AI safety researchers

• RohanS 8 Nov 2025 17:35 UTC

  5 points

  3

  in reply to: williawa’s comment on: 13 Ar­gu­ments About a Tran­si­tion to Neu­ralese AIs

  I think CoT may be very valuable in helping capabilities researchers understand their models’ shortcomings and design training curricula that address those deficiencies. This sort of falls under “business value” itself. I currently think this is one of the stronger arguments for why we won’t switch to neuralese.

  (The post only vaguely gestures at this argument, so your interpretation is also fair, but I think business value and capabilities gains are pretty closely related rather than being opposing factors.)

• RohanS 17 Oct 2025 21:13 UTC

  1 point

  0

  in reply to: Vladimir_Nesov’s comment on: Ro­hanS’s Shortform

  This veers into moral realism.

  That’s not an accident, I do lean pretty strongly realist :). But that’s another thing I don’t want to hardcode into AGIs, I’d rather maintain some uncertainty about it and get AGI’s help in trying to continue to navigate realism vs antirealism.

  I think I agree about the need for a morality-agnostic framework that establishes boundaries and coordination, and about the risks of dystopia if we attempt to commit to any positions on object-level morality too early in our process of shaping the future. But my hope is that our meta-approach helps achieve moral progress (perhaps towards an end state of moral progress, which I think is probably well-applied total hedonistic utilitarianism). So I still care a lot about getting the object-level moral considerations involved in shaping the future at some point. Without that, you might miss out on some really important features of great futures (like abolishing suffering).

  Perhaps relatedly, I’m confused about your last paragraph. If a single highly unusual person doesn’t conform to the kinds of moral principles I want to have shaping the future, that’s probably because that person is wrong, and I’m fine with their notions of morality being ignored in the design of the future. Hitler comes to mind for this category, idk what comes to mind for you.

  (I’ve always struggled to understand reasons for antirealists not to be nihilists, but haven’t needed to do so as a realist. This may hurt my ability to properly model your views here, though I’d be curious what you want your morality-agnostic framework to achieve and why you think that matters in any sense.)

  (I realize I’m saying lots of controversial things now, so I’ll flag that the original post depended relatively little on my total hedonistic utilitarian views and much of it should remain relevant to people who disagree with me.)

• RohanS 17 Oct 2025 18:23 UTC

  1 point

  0

  in reply to: Vladimir_Nesov’s comment on: Ro­hanS’s Shortform

  • I’m not so sure about total hedonistic utilitarianism that I want to directly stick it into future-shaping AIs, I’d rather have them “continuously work on a balance between figuring out what is best for the universe and acting according to [their] best current understanding of what is best for the universe (like I try to do)”

  • I think other people can be wrong about morality, in which case I don’t think their notion of a good future is something I need to try to promote

  • “Exploring or developing a philosophical position is distinct from espousing it.” If I understand correctly, this is mostly a matter of how well fleshed out I think the position is and how confident I am in it. I think there may be small pieces of the argument and the position that aren’t perfectly fleshed out, but I expect there to be ways to iron out the details, and I overall think the position and the argument are pretty well fleshed-out. I’d say I espouse total hedonistic utilitarianism (while also being open to further development).

  • If other people endorse other things and don’t consider hedonic valence important, then we should have good decision-making mechanisms for handling this kind of conflict as we shape the long-term future. I mentioned above (point 2 at the bottom of the original post) that I want such a decision process to be truth-seeking and prosocial. It should seek to figure out whether those other things actually matter and whether hedonic valence actually matters. If hedonic valence actually matters and nothing else does (as I suspect), then hedonic valence should be prioritized in decisions. In the prosocial part I’m including the idea that we should probably try ease the blow of this decision to deprioritize someone’s preferences. Maybe one example conflict is between factory farmers and people who want to eliminate factory farming. I’d want to eliminate factory farming, while offering new ways for the people currently reliant on factory farming to live good lives.

• RohanS 17 Oct 2025 15:25 UTC

  8 points

  1

  on: Ro­hanS’s Shortform

  What would it look like for AI to go extremely well?

  Here’s a long list of things that I intuitively want AI to do. It’s meant to gesture at an ambitiously great vision of the future, rather than being a precise target. (In fact, there are obvious tradeoffs between some of these things, and I’m just ignoring those for now.)

  I want AGI to:

  • end factory farming

  • end poverty

  • enable space exploration and colonization

  • solve governance & coordination problems

  • execute the abolitionist project

  • cultivate amazing and rich lives for everyone

  • tile a large chunk of the future lightcone with hedonium?

  • be motivating and fun to interact with

  • promote the best and strongest kinds of social bonds

  • figure out how consciousness works as well as possible and how to make it extremely positive

  • counteract climate change and racism and sexual abuse and sexism and crime and scamming and depression

  • make transportation amazing

  • make healthcare amazing (sickness, long-lasting body pain, aging, disease, cancer; speed and quality and cost of healthcare)

  • make incredible art and games and sports and get the right people to engage in them

  • make delicious food

  • take over menial tasks

  • maintain opportunities for immersion in natural beauty

  • figure out the right way to interact with aliens and interact with them that way (assuming we ever interact with aliens, which I think is quite plausible, even for generally intelligent ones)

  • maintain opportunities for intellectual stimulation and discovery?

  • defend against x-risks (other AGIs, nuclear, bio, climate, aliens)

  • reverse entropy?

  • help humans be happier w/​ less (e.g. via meditation)

  • satisfy Maslow’s full hierarchy of needs for everyone

  • improve wild animal welfare (drastically)

  • make moral progress consistently (if needed)

  • maintain human agency (where appropriate)

  • continuously work on a balance between figuring out what is best for the universe and acting according to its best current understanding of what is best for the universe (like I try to do)

  (Relevant context is that I’m a pretty confident total hedonistic utilitarian.)

  One possibly important takeaway is that a lot of this has to do with AI applications, and may continue to be hard to achieve with intent aligned AI for the same reasons they’re hard to achieve today: lack of consensus, going against local incentives for some powerful people and groups, etc. A couple (not novel) ideas for improving this:

  1. Maybe AI startups that work on these applications are actually counterfactually important

  2. Maybe we should try to make AIs be truth-seeking and prosocial in competitive envs (including when competing with each other), which may be better than e.g. the highly partisan US Congress as a mechanism for pursuing these goals on a large scale

• RohanS 1 Oct 2025 1:48 UTC

  1 point

  0

  on: Ro­hanS’s Shortform

  I was recently pleasantly surprised to discover the extent to which my short-term goals are aligned with my long-term goals. It’s a nice reminder that I’m working on the things I want to be working on.

  For an upcoming PhD lab meeting, I was asked to make a slide describing what success means to me on three different timescales, ~1, 5, and 15 years each. Setting aside the possibility of AGI within… any of those timescales, here’s what I wrote:

  ---

  What Is Success For Me?

  This Year: Excellent conference papers, blog posts about research results and AGI safety strategy, progress with Aether, connections (in the AIS community), research taste development, improvement at experiment velocity (incl. general coding and knowledge of tools) and writing. Influence AGI companies, governments, and/​or the AIS field in ways that later improve outcomes from AGI/​ASI.

  Short Term (PhD): Similar to the above :). I’m simultaneously aiming for direct impact, career capital, and reducing uncertainty about longer-term paths.

  Long Term: More of the above :). Open to industry (research scientist?), nonprofit, continuing to run Aether, starting other projects, academia, and even doing other useful things than ML research (like AGI strategy, grantmaking, policy, philosophy?, etc.).

  ---

  I can thank 80,000 Hours for the “direct impact, career capital, and reducing uncertainty” framing; I think it has done me a lot of good. I think 80k also helped me realize that the best way to do all three is to just directly try the first steps on one of your possible longer-term career paths, starting with cheaper tests and diving deeper into things that are going well. That certainly helped create the alignment I now see between my short and long term goals.

Hid­den Rea­son­ing in LLMs: A Taxonomy

How we spent our first two weeks as an in­de­pen­dent AI safety re­search group

Ex­tract-and-Eval­u­ate Mon­i­tor­ing Can Sig­nifi­cantly En­hance CoT Mon­i­tor Perfor­mance (Re­search Note)

• RohanS 13 Jul 2025 19:31 UTC

  4 points

  0

  in reply to: Gurkenglas’s comment on: Effi­ciently De­tect­ing Hid­den Rea­son­ing with a Small Pre­dic­tor Model

  It could be interesting to try that too, but we thought other reasoning models are more likely to predict similar things to the large reasoning models generating the CoTs in the first place. That hopefully increases the signal to noise ratio.