HoldenKarnofsky

• HoldenKarnofsky 16 Apr 2026 16:56 UTC

  4 points

  0

  in reply to: habryka’s comment on: Re­spon­si­ble Scal­ing Policy v3

  > The thing that I am comparing is “resources invested into advocating for direct regulation, and actions that would directly slow down AI development” vs. “resources invested into getting companies to adopt RSPs and get policies around RSPs passed”.
  
  This feels like an unfair comparison in that one of these things is much more specific/​narrow than the other.
  
  I think fairer comparisons would be:
  
  1. A comparison of two specific/​narrow goals along the lines of “Enact policy explicitly aimed at doing X.” Perhaps (a) “Advocating for pausing or stopping AI development, or perhaps for directly slowing it down via deliberate, explicit attempts to slow it down (not e.g. regulation largely or even putatively motivated by something else that happens to increase frictions to AI development)” vs. (b) “advocating for companies to adopt RSPs and get policies around RSPs passed.” I think SB53 and the EU Code of Practice have significant elements that look like the latter, whereas I’d guess we agree the former has gotten nowhere.
  
  2. A comparison of two broad approaches to spreading general messages or frameworks with many possible policy implications. Perhaps (a) “Advocating for direct regulation [which I assume refers to the sort of examples you gave before], and actions that would directly slow down AI development” vs. (b) “Advocating for the general paradigm of doing evaluations to assess current danger, implementing mitigations to reduce current danger, and having some sort of transparency and accountability around these activities.” Here too I think we’ve seen quite a bit more motion from the latter, both from legislation and via voluntary actions from companies, though perhaps you could frame a bit of this as the former, and if you have a certain picture of the risk, you may consider all of the motion on the latter to be worthless.
  
  > I agree and am deeply grateful for your work in the space, and your support of work in the space.
  
  I appreciate your saying so.

• HoldenKarnofsky 10 Apr 2026 4:21 UTC

  8 points

  3

  in reply to: habryka’s comment on: Re­spon­si­ble Scal­ing Policy v3

  Hm. I had thought you were pointing to something like “There isn’t actually going to be a pause in this environment triggered by if-then commitments” as the main update/​vindication of interest; I was basically responding by pointing out that there also isn’t going to be (and IMO there was never a promising path to) a pause in this environment triggered by advocacy for immediate pausing.
  
  Instead it seems like you’re doing something more like “comparing the overall impact of talking about pauses—or, more broadly, existential risks from AI—with the overall impact of talking about if-then commitments.” I think this is a much muddier comparison where there is less clearly a big update to be had.

  I don’t think we have seen much traction on attempts to slow down AI in any way. Meanwhile, I do think that the framework of “test for dangerous capabilities and implement commensurate mitigations” has had quite a significant impact on company behavior in a way that does seem to set up many policy possibilities that would otherwise be rough (including much of what has already passed).

  The comparison to “raise general awareness about risks of AI” (as opposed to “advocate for specific policies explicitly aimed at slowing down AI”) feels a bit harder to make—certainly I am, and long have been, positive disposed toward raising general awareness about risks of AI.
  
  But I will probably leave that there as it seems like a pretty complex and tricky debate to have.
  

  My understanding is that most safety efforts at Anthropic were oriented around the RSP in 2024/​2025, and my current model is that almost 50% of total safety talent in the ecosystem work at Anthropic.

  
  This doesn’t sound remotely right to me. I would say that the RSP has provided an organizing framework for a lot of safety work, but that’s different from something like “all of that safety work would make no sense if not for the RSP” or something.

• HoldenKarnofsky 12 Mar 2026 23:55 UTC

  15 points

  −16

  in reply to: aysja’s comment on: Re­spon­si­ble Scal­ing Policy v3

  It seems pretty misleading to describe the shift away from unilateral pausing as a natural extension of the RSP being a living document.

  You’re welcome to this view, but it isn’t mine. Other major projects I’ve worked on involved major, fundamental pivots that resulted in projects almost unrecognizable from the original pitch, and if you’d asked me at the time I would’ve said I expected something similar for RSPs. On priors, a completely new kind of risk management framework should be expected to change dramatically. I never would have agreed to “The policies we’re coming up with today will only change in the details.”

  I think doing so marks the breaking of a meaningful promise—something many people were relying on, making career decisions on the basis of, etc.

  I’ve seen many claims along these lines, and my guess is that there is at least some truth to them, but I am somewhat surprised by how little I’ve seen in the way of tangible specifics re: who promised what and who made important plans and decisions based on this. I have mostly just seen the quote from Evan, which Evan claims is a mischaracterization. I genuinely can’t recall encountering this sort of thing firsthand (though I’ve only been at Anthropic for about a year). Again, I’m not saying such things didn’t happen, but I haven’t seen enough specifics to be affirmatively convinced by claims like this or to have a clear sense of who was responsible, who was harmed, etc.

  In other words, because a pause would seriously damage the company, there was pressure to misrepresent the risk. I think this should seriously call Anthropic’s ability to self-govern into question

  I don’t think that feeling unhealthy pressures implies that there is a governance failure. For example, people regularly feel pressure to avoid admitting they were wrong—I don’t think this is a particular person’s fault or calls a particular governing structure into question. My statement here was about psychological pressures, not pressures imposed by e.g. executives.

  Which is to say that the situation as you’ve presented it seems strictly worse relative to the one Anthropic was imagining two years ago: we’re closer to AGI, but we have much less hope of accurately assessing the risk, and the political landscape is less favorable. Yet it seems like your proposal, in response an overall more dangerous situation, is to be even more reckless.

  I think the situation today is worse on the dimensions being discussed here (e.g., political will), although better on some other dimensions (IMO, the technical picture looks at least somewhat better).

  I don’t accept the framing that stricter commitments are necessarily less “reckless” while more flexible frameworks are more “reckless.” I think the new policy better positions us to reduce risk worldwide. If pausing were the only or clearly best path to risk reduction, I think it would make more sense to associate greater flexibility with greater recklessness. Perhaps you believe that’s the case; I don’t.

  But my god, does a post which is fundamentally premised on the inevitability of this race do so little to grapple with it. Not once does this post mention the possibility of extinction, for example, as if the real stakes and the real casualties Anthropic might cause have been forgotten. Very little attention is given to whether the race to AGI is in fact inevitable, or if there might be something Anthropic—as a leading player in this race (!)—might be able to do about that. Nor is any mention made of the role Anthropic has played in shaping this unfortunate political landscape which they now report being so helplessly beholden to. What is the point of having a seat at the table, if one doesn’t use it to wield influence in situations like this?

  I don’t believe a race is inevitable, and I think there may be things Anthropic can do to make a race less likely. But I think that those things are and will continue to be highly contingent on specific circumstances, and I also believe there are non-slowdown-oriented actions that can reduce risk significantly (and potentially more than slowdown-oriented actions, depending on the circumstances).

• HoldenKarnofsky 12 Mar 2026 23:53 UTC

  14 points

  −3

  in reply to: habryka’s comment on: Re­spon­si­ble Scal­ing Policy v3

  FWIW, my interpretation of what we should be learning is pretty different here.

  I would broadly say that political will for anything in the “slow down AI as needed to make it safe” category has been well short of what many people (such as myself) hoped for. Because of this, some of the core founding hopes of the RSP project look untenable now (although I don’t consider the matter totally settled); but to me it feels like an even bigger update away from “pause now” movements.

  I don’t understand why you say this: “the appetite for conditional risk regulation has been substantially less than the appetite for direct risk regulation (or compute thresholding, which doesn’t require any complicated eval infrastructure).” I have not seen roughly any appetite for “compute thresholding” if that means something like “limiting the size of training runs” (I have seen “compute thresholding” in the sense of “reporting requirements triggered by compute thresholds”). I don’t know what you mean by “direct risk regulation”, but if it means regulation aimed at slowing down AI immediately, I also have seen much less (roughly no) appetite/​momentum for that, and more for regulation based around things like evals and if-then commitments.

  Separately, with the benefit of hindsight, I think a global AI pause in 2023 would have been bad on the merits compared to, say, a pause around when the original RSP implied a pause should happen. The former, compared to the latter, would have meant losing a lot of opportunities for meaningful alignment research and more broadly for the world to learn important things relevant to AI safety, while having almost no marginal catastrophic risk reduction benefit AFAICT.

  You may have a view like “2023 was the right time to pause, because it was politically tractable then, but postponing it ensured it would not remain politically tractable.” That would be a very different read from mine on the political situation.

  > This is a huge deal! This was, as far as I can tell, the single decision that most affected talent allocation in the whole AI safety community. METR and Apollo and the broader “evals” agenda became the most popular and highest-prestige thing to work on for people in AI safety.

  This seems off to me. First, the emphasis on evals predated the idea of if-then commitments, and I think attracted more resources at pretty much every point in time; evals have a variety of potential benefits that don’t rely on if-then commitments. Second, I don’t think most people who work on AI safety work on either of these.

  What links here?

  • An­thropic Re­spon­si­ble Scal­ing Policy v3: A Mat­ter of Trust by Zvi (1 Apr 2026 18:10 UTC; 65 points)

• HoldenKarnofsky 12 Mar 2026 23:50 UTC

  11 points

  1

  in reply to: aysja’s comment on: Re­spon­si­ble Scal­ing Policy v3

  To me this language seems entirely consistent with my current position. In fact, if I had (at that time) been hoping for a voluntary pause, it would have been a strange choice for me to emphasis “consensus” leading to “regulation” when I could have just said something like “I’m excited about RSPs partly because it seems like people in those categories—not just people who agree with my estimates about risks—should support RSPs. This means that voluntary pausing based on conditional commitments is more likely than voluntary pausing today.”

  I believe that all of my past writing on if-then commitments presents the theory of change as running through policy action, and generally discusses “escape clauses” as well, rather than focusing on voluntary action as the theory of change.

• HoldenKarnofsky 12 Mar 2026 23:50 UTC

  17 points

  0

  in reply to: peterbarnett’s comment on: Re­spon­si­ble Scal­ing Policy v3

  Note this quote later in that same piece:

  Voluntary commitments and even regulation could be too hard to enforce across the board—such that responsible actors end up adhering to if-then commitments, while irresponsible actors rush forward with dangerous AI. One of the challenges with AI is that complete enforcement of any given risk mitigation framework seems extremely hard to achieve, yet incomplete enforcement could end up disadvantaging responsible actors in a high-stakes, global technology race. This is a general issue with most ways of reducing AI risks, other than “race forward and hope that the benefits outweigh the costs,” and is not specific to if-then commitments.

  To help mitigate this issue, early, voluntary if-then commitments can contain “escape clauses” along the lines of: “We may cease adhering to these commitments if some actor who is not adhering to them is close to building more capable models than ours.” (Some more detailed suggested language for such a commitment is provided by METR, a nonprofit that works on AI evaluations.)

Re­spon­si­ble Scal­ing Policy v3

• HoldenKarnofsky 27 Jan 2026 6:14 UTC

  4 points

  0

  in reply to: habryka’s comment on: An­thropic is (prob­a­bly) not meet­ing its RSP se­cu­rity commitments

  Nothing more to add for now, thanks for the response!

• HoldenKarnofsky 21 Jan 2026 4:20 UTC

  2 points

  0

  in reply to: habryka’s comment on: An­thropic is (prob­a­bly) not meet­ing its RSP se­cu­rity commitments

  There are a number of things you say here that don’t seem right to me and/​or aren’t capturing the intent of what I said. I prefer not to get into all of it, but just a couple of notes:

  • My current impression is that we are “highly protected against most attackers’ attempts at stealing model weights,” specifically highly protected against the groups listed as “in scope” (which I think of as including employees at partner orgs who have physical access to machines but not authorized access to weights), and broadly in line with the letter and spirit of the ASL-3 Security Standard. This isn’t my call and I am not up on all of the details of how we’ve vetted the security controls for partners, but it’s my impression.

  • An attacker being out of scope for the ASL-3 Security Standard does not meant “Anthropic wouldn’t consider it a security incident” if they stole (i.e., exfiltrated/​improperly used) important assets.

• HoldenKarnofsky 8 Jan 2026 17:53 UTC

  57 points

  5

  on: Con­tra­dict my take on OpenPhil’s past AI beliefs

  Some responses:

  RE: AGI ruin probability

  I agree with this comment from habryka re: what my view was for most of the relevant time period.

  If I were to give a similar range today, my top end would be lower (maybe something in the neighborhood of 50%?), so probably worse according to you than it was during the time period in question.

  RE: AGI timelines

  I think this post is probably the best public characterization of my state of mind about AI timelines as of November 2021, and broadly for several years before that too. The key part is here:

  • I would be at least mildly surprised if transformative AI weren’t developed by 2060. I put the probability of transformative AI by then at 50% (I explain below how the connection works between “mild surprise” and “50%”); I could be sympathetic to someone who said it was 25% or 75%, but would have a hard time seeing where someone was coming from if they went outside that range. More

  • I would be significantly surprised if transformative AI weren’t developed by 2100. I put the probability of transformative AI by then at 2 in 3; I could be sympathetic to someone who said it was 1 in 3 or 80-90%, but would have a hard time seeing where someone was coming from if they went outside that range. More

  • Transformative AI by 2036 seems plausible and concretely imaginable, but doesn’t seem like a good default expectation. I think the probability of transformative AI by then is at least 10%; I could be sympathetic to someone who said it was 40-50%, but would have a hard time seeing where someone was coming from if they said it was <10% or >50%. More

  I’m now at >50% by 2036, indeed by 2031 or so. So I think it’s plausible that this counts as “bad AI timelines,” depending on how foreseeable you thought the update was.

  In my defense on the narrow point: I know of very few people who made comparably specific statements contradicting the above in what looks like a more correct way. There were plenty of people expressing confidence in longer timelines; there were also a handful of people with shorter timelines, but some of the examples I can think of involve being too confident in the other direction. Eliezer made various statements implying short timelines (this is the best public example I have) but they tended to be vague and hard to interpret. I will credit Daniel Kokotajlo for his comment here, though—I now think he was more correct in that exchange.

  Good predictions vs. good decisions

  I also want to note that I find arguments of the form “X was doing a good/​bad job predicting parameter Y, and this is sufficient to establish that their decisions on this topic were good/​bad” generally uncompelling. It’s easy for me to think of people who made good (and seemingly well-reasoned in hindsight) decisions about some X while having mediocre or even bad predictions about it, or who made bad decisions while having excellent predictions.

  Predicting any given thing is a lot of work, and I often am trying to figure out things like “What’s the minimum amount of effort I can spend predicting specific parameter Y, while making good relevant decisions?” There are separate skills relating to taking actions that perform well under a wide variety of plausible scenarios, vs. brittle actions that are calibrated to specific predictions (even good ones). An analogy would be to a financial trader who cares a lot about whether they should buy or sell a stock, and what transactions they should make to hedge away unwanted risk, but doesn’t work nearly so hard on their full probability distribution over the future price of the stock.

  In this case:

  • A 10-50% chance of AGI by 2036 felt like plenty to motivate urgent action, and also felt like a big bet in the bullish direction relative to what most people implicitly thought.

  • It was very important to me that Open Philanthropy be taking the best opportunities conditional on particularly short timelines. I specifically spent significant time and energy on “What should we be doing if we have 10 years? 5 years? 3 years? Can we make sure we’re doing the top things we should be doing conditional on those assumptions?” between 2016-2022 when I was overseeing our work on AI risk.

  • I think if we had acted more as people with shorter timelines wanted us to act, this would’ve been worse in many cases than what we did. For example:

    • I think many of the people with the shortest timelines felt at the time that the primary implication of this was to more aggressively fund alignment research and put less effort into policy-related work.

    • There was also, IMO, a somewhat common vibe along the lines of “You should be treating grantmaking dollars out as a metric to be maximized; you should be ‘pushing on a string’ and aggressively seeing how much money you can shove out the door; anything that could plausibly absorb money should be generously funded.” I really don’t presently wish we had had more of this attitude, even if I assume we’re currently 1 year (or whatever) from AGI.

  None of this is to say that we didn’t make bad decisions! But if we did, I think it would make more sense to focus on what those were and why.

• HoldenKarnofsky 7 Jan 2026 0:16 UTC

  6 points

  0

  in reply to: habryka’s comment on: An­thropic is (prob­a­bly) not meet­ing its RSP se­cu­rity commitments

  I was in fact associating sophisticated insiders with actually having authorized access to model weights, and I’m not sure (even after asking around) why this is worded the way it is.

  I don’t really understand your comment here: “I don’t understand the relevance of this. Of course almost no one at the partners has “authorized” access to model weights. This is in the cybersecurity section of the RSP.” How many people have authorized access to a given piece of sensitive info can vary enormously (making this # no bigger than necessary is among the challenges of cybersecurity), and people can have authorized access to things that they are nevertheless not able to exfiltrate for usage elsewhere. It is possible to have very good protection against people with authorized access to model weights, and possible to have very little protection against this.

  My guess is that it is quite difficult for the people you’re gesturing at (e.g., people who can log in on the same machines but don’t have authorized access to model weights) to exfiltrate model weights, though I’m not personally confident of that.

• HoldenKarnofsky 5 Dec 2025 18:45 UTC

  2 points

  −2

  in reply to: habryka’s comment on: An­thropic is (prob­a­bly) not meet­ing its RSP se­cu­rity commitments

  Hi Oli, I think that people outside of the company falling under this definition would be outnumbered by people inside the company. I don’t think thousands of people at our partners have authorized access to model weights.

  I won’t continue the argument about who has an idiosyncratic reading, but do want to simply state that I remain unconvinced that it’s me (though not confident either).

• HoldenKarnofsky 22 Nov 2025 5:16 UTC

  12 points

  2

  in reply to: habryka’s comment on: An­thropic is (prob­a­bly) not meet­ing its RSP se­cu­rity commitments

  Thanks Oli. Your reading is quite different from mine. I just googled “insider risk,” clicked the first authoritative-ish-looking link, and found https://​​www.cisa.gov/​​topics/​​physical-security/​​insider-threat-mitigation/​​defining-insider-threats which seems to support something more like my reading.

  This feels like a quite natural category to me: there are a lot of common factors in what’s hard about achieving security from people with authorized access, and in why the marginal security benefits of doing so in this context are relatively limited (because the company has self-interested reasons to keep this set of people relatively contained and vetted).

  But it’s possible that I’m the one with the idiosyncratic reading here. My reading is certainly colored by my picture of the threat models. My concern for AIs at this capability level is primarily about individual or small groups of terrorists, I think security that screens off most opportunistic attackers is what we need to contain the threat, and the threat model you’re describing does not seem to me like it represents an appreciable increase in relevant risks (though it could at higher AI capability levels).

  In any case, I will advocate for the next iteration of this policy to provide clarification or revision to better align with what is (in my opinion) important for the threat model.

  FWIW, this is part of a general update for me that the level of specific detail in the current RSP is unlikely to be a good idea. It’s hard to be confident in advance of what will end up making the most sense from a risk reduction POV, following future progress on threat modeling, technical measures, etc., at the level of detail the current RSP has.

• HoldenKarnofsky 19 Nov 2025 20:48 UTC

  12 points

  3

  on: An­thropic is (prob­a­bly) not meet­ing its RSP se­cu­rity commitments

  Hi Oli, the threat model you’re describing is out of scope for our RSP, as I think the May 14 update (last page) makes clear. This point is separate from Jason’s point about security levels at cloud partners.

  (Less importantly, I will register confusion about your threat model here—I don’t think there are teams at these companies whose job is to steal from partners with executive buy-in? Nor do I think this is likely for executives to buy into in general, at least until/​unless AI capabilities are far beyond today’s.)

Sab­o­tage Eval­u­a­tions for Fron­tier Models

Case stud­ies on so­cial-welfare-based stan­dards in var­i­ous industries

Good job op­por­tu­ni­ties for helping with the most im­por­tant century

• HoldenKarnofsky 15 Jan 2024 20:57 UTC

  6 points

  0

  in reply to: simeon_c’s comment on: We’re Not Ready: thoughts on “paus­ing” and re­spon­si­ble scal­ing policies

  Thanks for the thoughts!

  #1: METR made some edits to the post in this direction (in particular see footnote 3).

  On #2, Malo’s read is what I intended. I think compromising with people who want “less caution” is most likely to result in progress (given the current state of things), so it seems appropriate to focus on that direction of disagreement when making pragmatic calls like this.

  On #3: I endorse the “That’s a V 1” view. While industry-wide standards often take years to revise, I think individual company policies often (maybe usually) update more quickly and frequently.

• HoldenKarnofsky 15 Jan 2024 20:54 UTC

  4 points

  0

  in reply to: aysja’s comment on: We’re Not Ready: thoughts on “paus­ing” and re­spon­si­ble scal­ing policies

  Thanks for the thoughts!

  I don’t think the communications you’re referring to “take for granted that the best path forward is compromising.” I would simply say that they point out the compromise aspect as a positive consideration, which seems fair to me—“X is a compromise” does seem like a point in favor of X all else equal (implying that it can unite a broader tent), though not a dispositive point.

  I address the point about improvements on the status quo in my response to Akash above.

• HoldenKarnofsky 15 Jan 2024 20:53 UTC

  14 points

  2

  in reply to: Orpheus16’s comment on: We’re Not Ready: thoughts on “paus­ing” and re­spon­si­ble scal­ing policies

  Thanks for the thoughts! Some brief (and belated) responses:

  • I disagree with you on #1 and think the thread below your comment addresses this.

  • Re: #2, I think we have different expectations. We can just see what happens, but I’ll note that the RSP you refer to is quite explicit about the need for further iterations (not just “revisions” but also the need to define further-out, more severe risks).

  • I’m not sure what you mean by “an evals regime in which the burden of proof is on labs to show that scaling is safe.” How high is the burden you’re hoping for? If they need to definitively rule out risks like “The weights leak, then the science of post-training enhancements moves forward to the point where the leaked weights are catastrophically dangerous” in order to do any further scaling, my sense is that nobody (certainly not me) has any idea how to do this, and so this proposal seems pretty much equivalent to “immediate pause,” which I’ve shared my thoughts on. If you have a lower burden of proof in mind, I think that’s potentially consistent with the work on RSPs that is happening (it depends on exactly what you are hoping for).

  • I agree with the conceptual point that improvements on the status quo can be net negative for the reasons you say. When I said “Actively fighting improvements on the status quo because they might be confused for sufficient progress feels icky to me in a way that’s hard to articulate,” I didn’t mean to say that there’s no way this can make intellectual sense. To take a quick stab at what bugs me: I think to the extent a measure is an improvement but insufficient, the strategy should be to say “This is an improvement but insufficient,” accept the improvement and count on oneself to win the “insufficient” argument. This kind of behavior seems to generalize to a world in which everyone is clear about their preferred order of object-level states of the world, and incentive gradients consistently point in the right direction (especially important if—as I believe—getting some progress generally makes it easier rather than harder to get more). I worry that the behavior of opposing object-level improvements on the grounds that others might find them sufficient seems to generalize to a world with choppier incentive gradients, more confusing discourse, and a lot of difficulty building coalitions generally (it’s a lot harder to get agreement on “X vs. the best otherwise achievable outcome” than on “X vs. the status quo”).

  • I think nearly all proponents of RSPs do not see them as a substitute for regulation. Early communications could have emphasized this point more (including the METR post, which has been updated). I think communications since then have been clearer about it.

  • I lean toward agreeing that another name would be better. I don’t feel very strongly, and am not sure it matters at this point anyway with different parties using different names.

  • I don’t agree that “we can’t do anything until we literally have proof that the model is imminently dangerous” is the frame of RSPs, although I do agree that the frame is distinct from a “pause now” frame. I’m excited about conditional pauses as something that can reduce risk a lot while having high tractability and bringing together a big coalition; the developments you mention are great, but I think we’re still a long way from where “advocate for an immediate pause” looks better to me than working within this framework. I also disagree with your implication that the RSP framework has sucked up a lot of talent; while evals have drawn in a lot of people and momentum, hammering out conditional pause related frameworks seems to be something that only a handful of people were working on as of the date of your comment. (Since then the number has gone up due to AI companies forming teams dedicated to this; this seems like a good thing to me.) Overall, it seems to me that most of the people working in this area would otherwise be working on evals and other things short of advocating for immediate pauses.