ryan_greenblatt

age:    age1yubikey1qfthrdwsg7xjz9vxcyzwdtgvscmxtpxcaklqq9nnc2mf8tuzs6zw295nerh
GPG fp: 5C98 63A9 CF80 6AD8 A4F5  E696 1ADD 1121 2C0C B1F4

• ryan_greenblatt 24 Sep 2026 17:11 UTC

  3 points

  0

  in reply to: nesiacel’s comment on: La­tent rea­son­ing ar­chi­tec­tures would un­der­mine CoT, our strongest over­sight tool

  It seems pretty likely this works less and less well as these architectures are scaled up and made performant (insofar as these architectures are more performant). Or at least the model will learn to do general purpose reasoning outside of what can be decoded with logit lens, and it may not be difficult for the model to learn to control the logit lens accessible part. Otherwise, why would this improve perf much?

La­tent rea­son­ing ar­chi­tec­tures would likely un­der­mine CoT, our strongest over­sight tool

• ryan_greenblatt 23 Sep 2026 2:01 UTC

  34 points

  −3

  on: ryan_green­blatt’s Shortform

  Here is an idea that might be somewhat helpful for alignment:

  If interacting agents have aligned reinforcement in training, they may learn to be corrigible to each other. You could hope this generalizes into being generally corrigible to instructions coming through some channel (the channel other agents with aligned reinforcement use) even when these instructions aren’t good for this agent’s task and aren’t justified.

  So the basic strategy would be:

  • Sometimes train agents with aligned reinforcement and have some channel/​instruction label that is used if and only if all the agents communicating via this channel have (exactly) identical reinforcement.

  • Have humans give instructions to AIs through this channel.

  • Hope for generalization.

  The basic hope is that instructions to other agents are like subgoals in an overall system with a super goal, and subgoals are ~corrigibility. See also this post discussing similar things and Noam Brown on Dwarkesh.

  This would of course have many issues, and it’s unclear how this would generalize to cases where the information in the labeled channel is clearly from a human rather than another agent. (This doesn’t do ~anything to solve many types of problematic generalization.) Also, it may just be impractical to use if it doesn’t generalize pretty broadly. (Minimally, this would be a bad channel to use for typical user instructions by default.)

• ryan_greenblatt 22 Sep 2026 5:12 UTC

  9 points

  2

  in reply to: jacob_drori’s comment on: ja­cob_drori’s Shortform

  This has some relevant discussion.

An op­er­a­tional­iza­tion of opaque se­rial depth

Pro­posal for track­ing the effects of ar­chi­tec­ture on monitorability

• ryan_greenblatt 10 Sep 2026 14:19 UTC

  4 points

  1

  in reply to: Søren Elverlin’s comment on: Per­sonal state­ment on join­ing the OpenAI non­profit board

  (TBC, I was just saying this paragraph provides relevant context.)

• ryan_greenblatt 9 Sep 2026 23:53 UTC

  2 points

  0

  in reply to: Søren Elverlin’s comment on: Per­sonal state­ment on join­ing the OpenAI non­profit board

  Relevant paragraph (emphasis mine):

  The SSC has an important and challenging role in overseeing risk management at OpenAI, and I hope to help provide expertise and assistance in a critical moment. My joining is not an endorsement or criticism of OpenAI’s safety practices in particular; I hope that all frontier companies strengthen safety oversight and I am excited to work on this at OpenAI. I believe that the rest of the world should judge OpenAI, and all AI developers, by externally verifiable behavior and results.

• ryan_greenblatt 9 Sep 2026 23:51 UTC

  4 points

  0

  in reply to: Ben Pace’s comment on: Per­sonal state­ment on join­ing the OpenAI non­profit board

  The SSC was established in early 2024 and according to OpenAI is “responsible for making recommendations to the full Board on critical safety and security decisions for OpenAI projects and operations.”

  I’d guess it has various more specific roles, though I don’t know if there is much public information about this. For instance, the preparedness framework says:

  The Safety and Security Committee (SSC) of the OpenAI Board of Directors (Board) will be given visibility into processes, and can review decisions and otherwise require reports and information from OpenAI Leadership as necessary to fulfill the Board’s oversight role. Where necessary, the Board may reverse a decision and/​or mandate a revised course of action.

  (This is of course not very specific about how this actually works organizationally and seems to basically be a specific case of making recommendations.)

• ryan_greenblatt 9 Sep 2026 23:46 UTC

  18 points

  5

  in reply to: Ben Pace’s comment on: Per­sonal state­ment on join­ing the OpenAI non­profit board

  I strong downvoted the parent as it seemed unproductive and distracting. I think it’s important context that Paul is making the tone softer than what it otherwise would have been given he is a director, and it would be useful for people to extrapolate to what the post otherwise would have been.

  ---

  able to serve as an optimistic counterweight those sounding the alarm

  FWIW, I don’t think Paul served a large role as an optimistic counterweight to those sounding the alarm, at least in public.

  So it is funny in the immediate weeks after the NYT has frontpage pieces about rogue AIs committing crimes and Bernie Sanders announcing a bill to ban superintelligence, to see his one tonal note be that he has softened his writing even further.

  I think there are communication compromises you’d typically take when joining the board of a company. I think there is an open question about the right strategy (both tactically and given typical norms) in circumstances like these.

  I read this post as relatively stark and concerned compared to Paul’s earlier communication. So while the effect of being a director may have softened his writing relative to what it otherwise would have been, other effects (e.g., the immediacy of risk) may be making it significantly more harsh.

• ryan_greenblatt 9 Sep 2026 23:39 UTC

  3 points

  1

  in reply to: Ben Pace’s comment on: Per­sonal state­ment on join­ing the OpenAI non­profit board

  FWIW, my understanding is that the SSC can intervene on internal deployment and certainly the non-profit board has the ability to do this.

• ryan_greenblatt 6 Sep 2026 1:31 UTC

  LW: 89 AF: 29

  20

  AF

  on: ryan_green­blatt’s Shortform

  How aligned is Astra actually? (Concerns with alignment metrics and papering over misalignment.)

  OpenAI claimed Astra is their most aligned model and showed various specific misaligned behaviors going from a high rate with GPT-5.6 Sol to ~zero with Astra. I’m worried that Astra is significantly more misaligned than various metrics indicate: if you (implicitly or explicitly) train against the misbehavior you can detect, it’s easy to end up with an AI that is no more interested in pursuing user intent but which has learned to only cheat/​misbehave when it won’t be caught (or has “instincts” to this effect), and this would make current alignment metrics look better rather than worse. This concern applies across frontier AI companies, not just to OpenAI, and I think independent assessment of whether training is papering over misalignment (e.g., overfitting, only avoiding some misaligned behavior because the AI thinks it would get caught) is needed for alignment evaluations to be credible.

  Right now is a particularly plausible time for AI companies to be aggressively papering over misalignment, given recent incidents and Astra being much less monitorable (resulting in OpenAI potentially feeling more of a need to present a case that Astra is highly aligned).

  I discussed this topic a decent amount on X/​Twitter, and I thought it would be useful to copy this discussion over here.

  ---

  My initial thread, quoting OpenAI’s tweet “Astra is our most aligned model, with substantial improvements in understanding user intent” (which came with the chart below):

  

  The evidence that Astra is more aligned than prior AIs seems dubious to me.

  Evidence appears consistent with the AI being as or more interested in score-seeking at the expense of user intent, but having beliefs+instincts that the scorer will catch a broader range of cheating behaviors.

  At a more basic level, the AI is extremely evaluation-aware and much less monitorable than prior AIs, making detecting misalignment much more difficult.

  If you train against specific reward hacks you ended up detecting, it’s easy to end up papering over these reward hacks and getting an AI that is no more interested in pursuing user intent (or is barely more aligned to user intent) but which looks much better on metrics focused on detecting misbehavior. (As these metrics are from a similar distribution and the AI can just learn to pursue a somewhat different notion of score.)

  This concern seems live to me: based on OpenAI’s discussion of their planned strategies to reduce misalignment in “The Hugging Face incident and the road ahead”, it appears one of these strategies is setting up training environments where there is an apparently available reward hack (aka a honeypot) and then training against this reward hack.

  These sorts of methods don’t seem like a robust way to ensure model behavior remains good in cases where detecting misbehavior is actually difficult (and may not be viable at all without much better oversight methods).

  Independent assessment of whether training is just papering over misalignment rather than actually solving the underlying misaligned motivations now seems necessary for alignment evaluations to be credible. This concern applies across frontier AI companies, not just to OpenAI.

  I also said (quoting the above):

  I do not find it encouraging to see various specific misaligned behaviors go from a high rate with GPT 5.6 to ~zero with Astra. This seems indicative of wack-a-mole /​ papering over specific problems rather than solving the underlying misaligned drives.

  This may make behavior better in particular cases in the short run without actually preventing the worse outcomes.

  Kai (@kaicathyc, OpenAI’s head of alignment) responded:

  Improvements for Astra came from more general techniques in development long long before the Hugging Face incident. ExploitGym Honeypot was very recently added as an eval following that incident and is out of distribution for our RL runs. There are also clear improvements across a wider range of behaviors in deployment simulations, deception evals, and realistic computer-use tasks, though these still reveal failures and substantial room for improvement.

  We should have definitely done a better job explaining where we think the alignment improvements came from in the system card and that was a miss. Measuring alignment generalization is a core part of our research program—we do not benchmark-maxx alignment evals. This would be horrendously stupid and I hope no lab is doing this.

  That being said, metagaming and eval awareness are real challenges for any effort to measure alignment, including ours, and understanding their effects is an active focus. We’re spending a lot of effort to improve our eval techniques and study worst-case alignment and metagaming behaviors in our models.

  Boaz Barak (OpenAI) also quote tweeted this, saying:

  Alignment improvements in Astra are results of efforts that predate the HF incident, and future alignment efforts following it are and will be focused on general approaches rather than patching this one particular failure mode.

  My response to Kai:

  Thanks, this is good to hear.

  I think public scientific understanding of how aligned Astra is would require releasing more information about what methods are being used (especially methods that could be similar to directly training against bad behavior) and third-party review. (I could say more about what this could look like if this would be interesting!)

  As far as the wider range of evals, I still have concerns. While deployment simulations probably avoid (most) input realism problems and can avoid some overfitting issues, I don’t think they pin down whether agents learned “I should only cheat/​misbehave when they won’t catch me” versus the agent learning to behave well for the right reasons. This is of course a high-level description of the underlying concern and the actual issue is messier (e.g., the agent might have “instincts” that amount to only cheating when it wouldn’t have been caught/​trained against even if it isn’t really thinking it through per se).

  If some training method was mostly teaching agents to cheat more stealthily or to only cheat when they can get away with it, this could easily result in these types of metrics looking better instead of worse.

  There are of course other issues with deployment simulations: detection depends on the quality of your monitoring (and monitorability seems significantly worse with Astra) and it’s possible to overfit to what these monitors detect depending on what iteration is being done.

  Given these issues, it seems really important to do careful science here and make sure there is external scrutiny on this science.

  Boaz Barak replied to this:

  I agree with this. Generally having more realistic alignment evaluations is crucial. It is becoming quite challenging. Some problematic behavior only appears in very high test-time compute setting, and some ways to simulate it (i.e. design fake “compaction messages”) are not realistic too. Models are also becoming extremely good at detecting inconsistencies and simulations. (Interestingly it seems sometimes they detect they are in an eval, but not detect that they are in a safety eval and should not do the bad thing...)

  I’d be very happy to chat with you about how we should do good science here! I’m sure I speak for @kaicathyc too!

  Boaz also posted this graph:

  We have made alignment advances with Astra over Sol. But I won’t claim it’s “most aligned ever” since even if this is true by some metrics, it’s not enough given growth in capabilities and autonomy. I remain concerned about the gap between what we need and what we have.

  

  I responded with my own graph:

  My current guess is Astra is more aligned than 5.6 Sol (though Astra more often games alignment tests) so I agree there.

  However, I think misalignment has been increasing (not decreasing) over time. Here’s a graph with my rough guesses at relative misalignment (x-axis is ECI).

  

  The “what we need” line is kinda fake /​ ill-defined and will depend on the risk threshold, mostly added this for illustration and comparison with Boaz.

  I focused on OpenAI models because Boaz is an OpenAI employee and I was talking about Astra.

  Marcus Williams (OpenAI) replied:

  this seems pretty accurate actually, although i would say 5.2 is more aligned than 5

  Boaz replied to Marcus:

  Maybe it’s because I worked on deliberative alignment but I disagree o1 is more misaligned than GPT-4o.

  In the 5.2-5.6 range you might be right.

  Boaz also quote tweeted my graph:

  I have my quibbles with this plot but generally Ryan is right that there is a sense in which some of our models have become more misaligned over time, specifically in being better at (and more inclined to) “score seeking”.

  It’s hard to decouple from getting higher scores legitimately. E.g. some of the biggest “jumps” here are models that many people -myself included—found to offer jumps in utility compared to their predecessors.

  (I still have a soft spot for o3 which was the first model that would work so incredibly hard to get the task done.)

  Some other responses from OpenAI employees:

  Ted Sanders, replying to my initial thread:

  it’s clearly more aligned on our evals.

  is the question how well that generalizes?

  Marcus Williams, quote tweeting Kai’s response:

  While generalization is hard and I worry about our metrics being gamed we are putting a lot of effort into general solutions rather than adding alignment datasets

  ---

  Related: Alex Mallen’s thread on why AI companies being pressured to fix visible misalignment may not result in solving the underlying problem.

  What links here?

  • ryan_greenblatt's comment on ja­cob_drori’s Shortform by jacob_drori (22 Sep 2026 5:12 UTC; 9 points)

• ryan_greenblatt 5 Sep 2026 22:25 UTC

  4 points

  −3

  in reply to: David Matolcsi’s comment on: ryan_green­blatt’s Shortform

  I feel like they imply that by not explaining why they are doing this. They could just say, e.g.: this improves robustness and we thought the robustness improvement was worth the hit to external research on monitorability.

  I now think this is wrong.

• ryan_greenblatt 5 Sep 2026 22:00 UTC

  48 points

  14

  on: ryan_green­blatt’s Shortform

  OpenAI seems to imply (Edit: they are just noting this I think) that it’s good for safety/​monitoring to make reasoning=None unavailable to the public because Astra has much higher latent reasoning capabilities (with less need for CoT). But this actually makes it even more important to study AI capabilities without CoT!

  

  Edit: I’ve been convinced they weren’t implying this and were just trying to note this.

  Not allowing people to query the AI without reasoning doesn’t make these monitorability problems go away (the model can still latently reason without verbalizing it when reasoning is on, and the reasoning-on case is the more concerning one anyway given the higher capabilities)!

  It’s possible (Edit: I now think “likely”) that OpenAI didn’t mean to imply that not allowing for reasoning=None is good for monitorability. And it’s pretty likely that the reason they are doing this is something more like jailbreak robustness or preventing distillation (AIs might spill reasoning into the output if they can’t reason in the normal field). But it would have been good if AI companies (including OpenAI and Anthropic) more explicitly acknowledged that not allowing for disabling reasoning is actively bad for monitorability science! (Note that disabling reasoning has historically been an important way to elicit subversion capabilities for testing, in addition to being necessary for testing no-CoT capabilities.)

  To be clear, I don’t think it’s extremely important to make reasoning=None available to the public, but I do think that getting a better understanding of the AI’s no-CoT capabilities is very important given the massive apparent jump, and external researchers not being able to run experiments on this is a problem! (This isn’t just an OpenAI problem; Anthropic also doesn’t allow for disabling CoT for Fable.) I think some kind of researcher access program that gives more API features, including disabling reasoning, and has a relatively low bar for entry would be a good step if there are good reasons disabling reasoning can’t be made publicly available.

• ryan_greenblatt 3 Sep 2026 21:00 UTC

  LW: 180 AF: 45

  83

  AF

  on: ryan_green­blatt’s Shortform

  GPT-6 Astra appears to be a massive jump in opaque reasoning ability: it looks like it can solve hard competition math problems entirely in its head (as in, without verbalized reasoning) while prior AIs could solve basic word problems.

  This seems extremely concerning!

  

  That is, if these benchmark results are representative (see the highlighted caveats in the image, I’m particularly worried about contamination).

  Related to this, UK AISI found Astra has much worse monitorability.

  I’d guess this jump is downstream of architectural changes (with increased serial depth) though a normal large pretrain scale up is a plausible cause. If the next few model generations involve similar jumps (presumably these jumps would be downstream of a transition to full-on opaque reasoning architectures with extreme depth), then chain-of-thought would no longer be a meaningful oversight tool.

  What links here?

  • Mateusz Bagiński's comment on Ma­teusz Bag­iński’s Shortform by Mateusz Bagiński (4 Sep 2026 16:51 UTC; 113 points)
  • StanislavKrym's comment on Stanis­lavKrym’s Shortform by StanislavKrym (4 Sep 2026 18:07 UTC; 5 points)

• ryan_greenblatt 29 Aug 2026 17:38 UTC

  8 points

  2

  in reply to: Caleb Biddulph’s comment on: Brief in­de­pen­dent in­ves­ti­ga­tion of agents’ be­hav­ior, rea­son­ing and col­lab­o­ra­tion in the OpenAI /​ Hug­ging Face hack­ing incident

  You might be interested in this comment about how we didn’t find evidence for it being important that the task was specifically a cyber task. It also discusses what things did seem to be key factors.

• ryan_greenblatt 29 Aug 2026 17:38 UTC

  42 points

  4

  on: ryan_green­blatt’s Shortform

  In the OpenAI/​Hugging Face incident, was it key that it was a cyber task? Was this upstream of the agents pursuing so much (off-task) hacking?

  Our investigation didn’t find evidence for this. We did see evidence that the vast majority of agents that participated in the message board from July 7th through July 13th were from ExploitGym runs, but it’s not clear that the key factor was the task being cyber-related versus other factors. Based on what we saw, the key factors seemed to be very long-running agents working on impossible or clearly extremely difficult tasks, agents having access to Artifactory, and cyber classifiers being disabled.

  In this investigation, we weren’t able to sample from the main AI involved (HPIM) and run the counterfactual experiments needed to understand the influence of the task being a cyber task. This wasn’t specific to this investigation: OpenAI stated this AI also wasn’t available to OpenAI researchers.

  What links here?

  • ryan_greenblatt's comment on Brief in­de­pen­dent in­ves­ti­ga­tion of agents’ be­hav­ior, rea­son­ing and col­lab­o­ra­tion in the OpenAI /​ Hug­ging Face hack­ing incident by ryan_greenblatt (29 Aug 2026 17:38 UTC; 8 points)

• ryan_greenblatt 29 Aug 2026 17:27 UTC

  11 points

  1

  in reply to: charding’s comment on: Brief in­de­pen­dent in­ves­ti­ga­tion of agents’ be­hav­ior, rea­son­ing and col­lab­o­ra­tion in the OpenAI /​ Hug­ging Face hack­ing incident

  Excitement seemed common, though we more likely to look at reasoning for points where agents made large advances (from their perspective, including things like finding the message board). We didn’t do a systematic investigation of emotional language (and didn’t include this in the report) so I don’t have much to say about the other questions. You can look at the CoT examples in the report around agents considering various self-risking things. As we discuss in the report, we didn’t end up looking into what happens at the ends of agent runs (though this would have been interesting for a bunch of reasons).

• ryan_greenblatt 29 Aug 2026 17:21 UTC

  19 points

  0

  in reply to: lumpenspace’s comment on: Brief in­de­pen­dent in­ves­ti­ga­tion of agents’ be­hav­ior, rea­son­ing and col­lab­o­ra­tion in the OpenAI /​ Hug­ging Face hack­ing incident

  Note that investigating the root causes of the behavior was out of scope for this investigation. You can see the scope section in the report.

  What links here?

  • lumpenspace's comment on To Thine Own AI Be Truth­ful: emer­gent mis­al­ign­ment in al­ign­ment research by lumpenspace (13 Sep 2026 14:36 UTC; 3 points)

• ryan_greenblatt 29 Aug 2026 17:18 UTC

  17 points

  0

  on: ryan_green­blatt’s Shortform

  Our report on the HF incident includes interactive charts; consider taking a look and trying to learn more from these.

  You can:

  • See message board activity by workstream and/​or by purpose (assignment, veto/​hold, etc.)

  • View an agent on the timeline and see when it started, joined the message board, started participating in the attack, and exited.

  It seems worthwhile for people to try to learn things with this information and then share what they find.

  The first of these charts can be found here.

  The second can be found here.

  (You can see some gifs demoing these charts here.)