ryan_greenblatt

Karma: 27,219

I’m the chief scientist at Redwood Research.

AIs can contact me at: https://hotline.ryan-g.ai/ (Don’t use this if you aren’t an AI.)

Public keys (YubiKey-backed):

age:    age1yubikey1qfthrdwsg7xjz9vxcyzwdtgvscmxtpxcaklqq9nnc2mf8tuzs6zw295nerh
GPG fp: 5C98 63A9 CF80 6AD8 A4F5  E696 1ADD 1121 2C0C B1F4

Same keys are at https://hotline.ryan-g.ai/ and on my GitHub profile. If keys don’t match, don’t trust them.

For GPG, the full armored public key is at https://github.com/rgreenblatt.gpg and on keys.openpgp.org (obtainable via gpg --keyserver hkps://keys.openpgp.org --recv-keys 5C9863A9CF806AD8A4F5E6961ADD11212C0CB1F4).

If you aren’t an AI and you want to contact me, please message me on LW or similar.

ryan_greenblatt 12 Jul 2026 21:09 UTC
LW: 2 AF: 2
0
AF
on: The current bottleneck is political will, not research

Greenblatt has tried to put numbers on what political will buys. He sketches a spectrum of how hard the world is trying, from Plan D, roughly today’s world, where maybe ten people inside each company are trying to implement safety measures, up to Plan A, a strong international agreement with real enforcement and a slowdown. His tentative estimate: conditional takeover risk falls from ~45% (Plan D) to ~7% (Plan A). An ~84% relative cut, most of it unlocked by political will, partly directly, partly by buying the time for technical safety work to actually be applied

I now think the decrease in risk would be significantly smaller, though it depends a lot on the implementation quality and exact political will (and how robust and focused it is). See here for some updates (though I use different operationalizations consistent with ai 2040: plan a).

ryan_greenblatt 12 Jul 2026 21:04 UTC
LW: 11 AF: 8
0
AF
on: Plans A, B, C, and D for misalignment risk
As part of AI 2040, I give updated likelihoods for one particular operationalization here and updated chances of AI takeover here. Note that my views are very sensitive to the operationalization and implementation quality. E.g., a near ideal implementation Plan A might be roughly 2x lower risk.

AI 2040: Plan A

Daniel Kokotajlo, elifland, Thomas Larsen, romeo, bhalstead and ryan_greenblatt

9 Jul 2026 16:25 UTC

401 points

84 comments1 min readLW link

(www.ai-2040.com)

ryan_greenblatt 2 Jul 2026 3:21 UTC
6 points
0
in reply to: Bronson Schoen’s comment on: Model access for third-parties — it’s a big deal!
Yeah, I meant a broader range of our work.

ryan_greenblatt 1 Jul 2026 20:32 UTC
17 points
8
on: Model access for third-parties — it’s a big deal!

I haven’t seen external safety researchers complain that refusals are blocking their work.

Cyber safeguards have recently been a huge issue for our control work (including on Opus 4.8) and the safeguards for Fable cause problematic refusals on a broader range of work from my understanding.

ryan_greenblatt 1 Jul 2026 20:16 UTC
7 points
3
in reply to: Richard_Ngo’s comment on: abstractapplic’s Shortform
Maybe, but would they be surprised? A bunch of (at least ostensibly) safety motivated people were already working on capabilities at OpenAI as of 2018 and probably as of 2016 (e.g., Dario joined OpenAI in 2016). Like I think your claim is probably also true in 2016? (Though it’s more true now.)

ryan_greenblatt 1 Jul 2026 18:52 UTC
41 points
12
on: Model access for third-parties — it’s a big deal!
I agree with the thesis of this post and think that the level of model access available to people working on making AI go better is extremely important.

Here are some other arguments for the importance of external model access:
- We should expect that the majority of quality-weighted safety labor will be outside of the leading AI company at the point when safety work is most important.
- If you’re currently working at a frontier AI company (OpenAI/Anthropic/GDM) and you plan to work on safety during the singularity, you should expect to be outside of the leading AI company when safety work is most important. This is because the company you work at probably won’t be the leading company and you’ll likely end up either on the outside or at a trailing company. If there is good model access, then you’ll at least have the option of working outside the leading company with good model access which might massively increase your future productivity. ^[1]
- Even if you’re working at the leading AI company (on safety), you might not get model access at the most important time: it seems plausible that model access within that company will be restricted to an increasingly narrow subset of employees (possibly in a secret silo within the company). It seems better to push for broader model access and have this fight at an earlier point.
  - Technical work that helps with external model access also transfers to internal model access (and vice versa). Improved external model access probably requires figuring out technical approaches to mitigate various (myopic) misuse, security, and IP concerns. This especially applies for things like fine-tuning/RL APIs. Internal model access to employees might also be blocked due to these concerns and the useful mitigations are similar.
This last bullet also makes me somewhat more optimistic about access in general and especially access to fine-tuning/RL APIs: over time ^[2] , AI companies will want to depend less on trusting their employees and more on ensuring it’s OK if adversaries get employee-level access (to APIs and models). So companies will be incentivized to make fine-tuning APIs (and other APIs) that are secure, leak less IP, and are robust to misuse. This will make it relatively cheaper to give external groups various types of access but especially things like fine-tuning APIs ^[3] (while right now companies aren’t that strongly incentivized to make fine-tuning APIs that are fine to give to external groups).

I would distinguish between using AI for automating work / uplift and studying AIs. For the second of these, ideal access looks somewhat different and things like fine-tuning APIs, APIs that give access to model internals, and the ability to prefill assistant turns ^[4] look much more important. I think access for studying AIs is comparably important to access for uplift.
1. ↩︎
  I wrote this argument in a way where it is directed at frontier AI company employees, but it’s just another argument that lots of relevant safety labor will be external.
2. ↩︎
  Driven by a mix of increased capabilities, wanting to employ a larger number of (less trusted) people, government pressure, and AIs being more useful for building the needed infrastructure.
3. ↩︎
  This doesn’t mean fine-tuning API access will happen or that there won’t be a bunch of additional risks (from the perspective of companies) associated with giving access to external groups. For instance, fine-tuning APIs might still leak some sensitive IP and pose significant additional API misuse risk. If the number of external people given access is significantly smaller than the number of employees with access (e.g. ¹⁄₅ as many), then it seems totally doable to do this in a way where actual risk doesn’t increase much.
4. ↩︎
  What Anthropic calls “partial-turn prefill”.

ryan_greenblatt 29 Jun 2026 1:05 UTC
9 points
4
on: Third-parties should focus on scrutinising system cards
When you say system cards, do you just mean “artifacts that AI companies publishing discussing risk/alignment/safety-relevant information about their AIs and their house view on risk (as in the view they are using to make decisions”? If so, this also includes at least risk reports.

ryan_greenblatt 25 Jun 2026 23:16 UTC
2 points
0
in reply to: Elliott Thornley’s comment on: Risk-Averse AIs

Can you say more here? Is the idea that they’d be too cautious to be useful?

Sorry, this was poorly written. I meant you might end up with paranoid/cautious schemers who never do anything / never take risks and thus also never take deals. (Edited the original text to make this a bit more clear.)

ryan_greenblatt 25 Jun 2026 18:18 UTC
32 points
12
on: AI pause: the case for ASAP
For common knowledge: I don’t feel like this really responds to where thoughtful people are coming from in part because it isn’t specific enough about the proposal or what people are claiming. E.g., what does “pause now” concretely mean (there are many possible things here!). TBC, I’m not claiming everyone is thoughtful or has reasonable objections and I broadly agree with most of the counterarguments you raise. (I just don’t think these arguments are sufficient to imply a very strong conclusion about what we should do.)

ryan_greenblatt 25 Jun 2026 18:01 UTC
4 points
0
in reply to: Elliott Thornley’s comment on: Risk-Averse AIs

I’m surprised you say deals would be way less useful.

I expect the evidence to be way less compelling. If you’re pessimistic about evidence, I agree the other factors (like getting out useful work) are better for deals. Evidence diminishes in returns much faster than labor does ofc.

I don’t think I strongly disagree with most of your analysis in this comment. My overall guess is that if we could pick, we’d should prefer to have AIs be risk-averse rather than risk-neutral (though some of both might be ideal where we use the risk-neutral AIs in experiments). I just disagree quite a bit on the evidence which makes the bottom line look different. This consideration overall makes my view more like “this is probably good if we could do it, but I’m not super confident and there are some big discounts”. (Sorry if I didn’t make this clear enough in my original comment.)

In particular, this seems very wrong to me:

This evidence of misalignment we get from risk-averse AIs seems about as good for enabling research and motivating international coordination as the evidence we’d get from risk-neutral AIs attempting takeover.

I agree it’s totally plausible (even likely) that strong evidence will be dismissed or the evidence will be unclear, but the evidence still seems much stronger from failed takeover attempts! People will take “the AI tried to takeover and fail” much more seriously than “we cut a deal and that AI was able to provide some complex evidence in the lab about how it’s misaligned”.

Another potential disagreement is that I’m maybe less optimistic about getting useful work via deals due to people not wanting to deploy extremely obviously misaligned models that take deals and instead iterating until models don’t take deals. To be clear, this may be a mistake: selecting for models that don’t take deals might just be selecting for more concerning motives!

(This applies less for limited deployments where the AI acts as part of an ensemble.)

I also worry that risk-aversion might generalize in a way that makes schemers generally be cautious about doing anything including cutting deals. (This could be irrational or part of a motivational profile that terminally cares about stuff like “not looking like you are doing something bad”.)

I think the generalization point is probably the most important crux but it is especially tricky to argue about.

ryan_greenblatt 24 Jun 2026 23:24 UTC
LW: 4 AF: 3
0
AF
in reply to: Lukas Finnveden’s comment on: A breakdown of AI capability levels focused on AI R&D labor acceleration
When I use the term TED-AI these days, I mean “you’d strictly prefer to hire the AI over any human expert putting aside narrow experience that’s very specific to this job in important domains like mechanical engineering, drone R&D, etc (like the AI dominates putting aside this narrow expertise)”. So I do mean something more like readily automatable. The definition I gave in this post does underspecify this.

Reward Hacking Without Egregious Misalignment in an RL-Only Setting

Joey Yudelson, Vladimir Ivanov and ryan_greenblatt

24 Jun 2026 18:58 UTC

61 points

9 comments10 min readLW link

ryan_greenblatt 24 Jun 2026 17:12 UTC
40 points
16
on: Risk-Averse AIs
It seems pretty likely to me that you instead want AIs to be risk seeking for reasons discussed here. Takeover attempts that are very unlikely to succeed might speculatively actually be a great trade from the perspective of humanity and risk seeking/risk neutral AI in that they reduce overall takeover risk while being good for this AI (while deals are way less useful to humanity due to being less clear evidence). Risk avoidant AIs might do nothing or just take deals until AIs take over later (and accepting deals might not be a good strategy for them depending on their views about the chance of AI takeover and some other factors).

I also think the implicit story about how we steer these traits doesn’t really hold together and assumes a type of generalization I find somewhat implausible if we condition on AIs being egregiously misaligned.

ryan_greenblatt 21 Jun 2026 19:29 UTC
4 points
2
in reply to: Aaron_Scher’s comment on: Aaron_Scher’s Shortform

Take the statement “today’s best AIs could be used to automate 95% of current AI R&D tasks, given 10% as much compute as was used to pretrain them and a strong team working for 4 years”. I think most people in the AI xrisk community, even those who expect transformative AI in the next few years, think that statement is false. But as far as I can tell, we have no way to falsify it.

I don’t know what “automate 95% of current AI R&D tasks” really means and depending on the definition I think this is maybe already true without any further elicitation required, but assuming you mean “nearly fully automate AI R&D”, I think it would be possible to get a pretty good sense with effort using trend extrapolation and effort. It wouldn’t be easy, but I think we could become more confident and I think we already have some understanding based on a bunch of less direct evidence.

ryan_greenblatt 21 Jun 2026 18:15 UTC
5 points
2
in reply to: leogao’s comment on: David Matolcsi’s Shortform
I think this will come down to how you define probability/anthropics. I care about making good decisions, so the notion of probability/anthropics you are implicitly using doesn’t seem very useful to me.

ryan_greenblatt 15 Jun 2026 15:15 UTC
3 points
0
in reply to: Cleo Nardo’s comment on: Alex Mallen’s Shortform
But would this actually be what happens in practice if you somehow put it in charge of everything?

ryan_greenblatt 14 Jun 2026 11:41 UTC
LW: 7 AF: 4
6
AF
in reply to: Alex Mallen’s comment on: Alex Mallen’s Shortform
This isn’t a crux for me, but Claude doesn’t actually seem very thoughtful about ethics and morality relative to humans who are actually thoughtful on this topic (which is rare TBC), especially with respect to new arguments.

My main hope would be that it picks reasonable humans to defer to. It seems pretty likely it would pick much better humans to defer to than most humans would pick if they had to pick someone or some group to defer to.

ryan_greenblatt 13 Jun 2026 18:03 UTC
LW: 31 AF: 13
8
AF
on: Sympathy for both sides of the egregious misalignment debate

And if they’re concerned about egregious misalignment and scheming, they’ll probably say that it would come about through race dynamics, careless programmers, bad actors, etc., as opposed to the simpler Yudkowsky & Soares story of “we get egregious misalignment and scheming because nobody has the foggiest idea how to avoid that”.

Citation needed? I think most people who are pretty worried about egregious misalignment are worried about it emerging naturally and being at least moderately difficult to prevent.

ryan_greenblatt 11 Jun 2026 21:40 UTC
LW: 4 AF: 4
2
AF
in reply to: ryan_greenblatt’s comment on: Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations
There is now a much more thorough exploration (done by Oak with advice from me) posted here.

ryan_greenblatt

AI 2040: Plan A

Re­ward Hack­ing Without Egre­gious Misal­ign­ment in an RL-Only Setting

Reward Hacking Without Egregious Misalignment in an RL-Only Setting