Evan Hubinger (he/him/his) (evanjhub@gmail.com)
Head of Alignment Stress-Testing at Anthropic. My posts and comments are my own and do not represent Anthropic’s positions, policies, strategies, or opinions.
Previously: MIRI, OpenAI
See: “Why I’m joining Anthropic”
Selected work:
Link is here, if anyone else was wondering too.
I think trying to make AIs be moral patients earlier pretty clearly increases AI takeover risk
How so? Seems basically orthogonal to me? And to the extent that it does matter for takeover risk, I’d expect the sorts of interventions that make it more likely that AIs are moral patients to also make it more likely that they’re aligned.
I think the most plausible views which care about shorter run patienthood mostly just want to avoid downside so they’d prefer no patienthood at all for now.
Even absent AI takeover, I’m quite worried about lock-in. I think we could easily lock in AIs that are or are not moral patients and have little ability to revisit that decision later, and I think it would be better to lock in AIs that are moral patients if we have to lock something in, since that opens up the possibility for the AIs to live good lives in the future.
I think it’s better to focus on AIs which we’d expect would have better preferences conditional on takeover
I agree that seems like the more important highest-order bit, but it’s not an argument that making AIs moral patients is bad, just that it’s not the most important thing to focus on (which I agree with).
Certainly I’m excited about promoting “regular” human flourishing, though it seems overly limited to focus only on that.
I’m not sure if by “regular” you mean only biological, but at least the simplest argument that I find persuasive here against only ever having biological humans is just a resource utilization argument, which is that biological humans take up a lot of space and a lot of resources and you can get the same thing much more cheaply if you bring into existence lots of simulated humans instead (certainly I agree that doesn’t imply we should kill existing humans and replace them with simulations, though, unless they consent to that).
And I think even if you included simulated humans in “regular” humans, I also think I value diversity of experience, and a universe full of very different sorts of sentient/conscious lifeforms having satisfied/fulfilling/flourishing experiences seems better than just “regular” humans.
I also separately don’t buy that it’s riskier to build AIs that are sentient—in fact, I think it’s probably better to build AIs that are moral patients than AIs that are not moral patients.
I mostly disagree with “QoL” and “pose existential risks”, at least in the good futures I’m imagining—those things are very cheap to provide to current humans. I could see “number” and “agency”, but that seems fine? I think it would be bad for any current humans to die, or to lose agency over their current lives, but it seems fine and good for us to not try to fill the entire universe with biological humans, and for us to not insist on biological humans having agency over the entire universe. If there are lots of other sentient beings in existence with their own preferences and values, then it makes sense that they should have their own resources and have agency over themselves rather than us having agency over them.
I really don’t understand this debate—surely if we manage to stay in control of our own destiny we can just do both? The universe is big, and current humans are very small—we should be able to both stay alive ourselves and usher in an era of crazy enlightened beings doing crazy transhuman stuff.
To be clear: we are most definitely not yet claiming that we have an actual safety case for why Claude is aligned. Anthropic’s RSP includes that as an obligation once we reach AI R&D-4, but currently I think you should read the model card more as “just reporting evals” than “trying to make an actual safety case”.
It was always there.
They’re releasing Claude Code SDK so you can use the core agent from Claude Code to make your own agents (you run /install-github-app within Claude Code).
I believe the Claude Code SDK and the Claude GitHub agent are two separate features (the first lets you build stuff on top of Claude Code, the second lets you tag Claude in GitHub to have it solve issues for you).
If Pliny wants jailbreak your ASL-3 system – and he does – then it’s happening.
Or rather, already happened on day one, at least for the basic stuff. No surprise there.
Unfortunately, they missed at least one simple such ‘universal jailbreak,’ that was found by FAR AI in a six hour test.
From the ASL-3 announcement blog post:
Initially [the ASL-3 deployment measures] are focused exclusively on biological weapons as we believe these account for the vast majority of the risk, although we are evaluating a potential expansion in scope to some other CBRN threats.
So, none of the stuff Pliny or FAR did is actually in scope for our strongest ASL-3 protections right now, since the Pliny and FAR attacks were for chem and we are currently only applying our strongest ASL-3 protections for bio.
So what’s up with this blackmail thing?
We don’t have the receipts on that yet
We should have more to say on blackmail soon!
The obvious problem is that 5x uplift on 25% is… 125%. That’s a lot of percents.
We measure this in a bunch of different ways—certainly we are aware that this particular metric is a bit weird in the way it caps out.
Auditors find an issue, and your reaction is that “Oh we forgot to fix that, we’ll fix it now”? I’ve participated in IT system audits (not in AI space), and when auditors find an issue, you fix it, figure out why it occurred in the first place, and then you re-audit that part to make sure the issue is actually gone and the fix didn’t introduce new issues. When the auditors find only easy-to-find issues, you don’t claim the system has been audited after you fix them. You worry how many hard-to-find issues were not found because the auditing time was wasted on simple issues.
Anthropic’s RSP doesn’t actually require that an external audit has greenlighted deployment, merely that external expert feedback has to be solicited. Still, I’m quite surprised that there are no audit results from Apollo Research (or some other organization) for the final version.
How serious do you think the issue is that Apollo identified? Certainly, it doesn’t seem like it could pose a catastrophic risk—it’s not concerning from a biorisk perspective if you buy that the ASL-3 defenses are working properly, and I don’t think there are really any other catastrophic risks to be too concerned about from these models right now. Maybe it might try to incompetently attempt internal research sabotage if you accidentally gave it a system prompt you didn’t realize was leading it in that direction?
Generally, I think it just seems to me like “take this very seriously and ensure you’ve fixed it and audited the fix prior to release because this could be dangerous right now” makes less sense as a response than “do your best to fix it and publish as much as you can about it to improve understanding for when it could be dangerous in smarter models”.
Why in the world would you use ARC-AGI to measure coding performance? It’s really a pattern-matching task, not a coding task. Also more here:
ARC-AGI probably isn’t a good benchmark for evaluating progress towards TAI: substantial “elicitation” effort could massively improve performance on ARC-AGI in a way that might not transfer to more important and realistic tasks. I am more excited about benchmarks that directly test the ability of AIs to take the role of research scientists and engineers, for example those that METR is developing.
I agree that the current thresholds and terminology are confusing, but it is definitely not the case that we just dropped ASL-4. Both CBRN-4 and AI R&D-4 are thresholds that we have not yet reached, that would mandate further protections, and that we actively evaluated for and ruled out in Claude Opus 4.
This is false. Our ASL-4 thresholds are clearly specified in the current RSP—see “CBRN-4” and “AI R&D-4″. We evaluated Claude Opus 4 for both of these thresholds prior to release and found that the model was not ASL-4. All of these evaluations are detailed in the Claude 4 system card.
See also some notes on reward hacking on twitter and in the model card.
I agree that attending an event with someone obviously shouldn’t count as endorsement/collaboration/etc. Inviting someone to an event seems somewhat closer, though.
I’m also not really sure what you’re hinting at with “I hope you also advocate for it when it’s harder to defend.” I assume something about what I think about working at AI labs? I feel like my position on that was fairly clear in my previous comment.
To be clear, I’m responding to John’s more general ethical stance here of “working with moral monsters”, not anything specific about Cremieux. I’m not super interested in the specific situation with Cremieux (though generally it seems bad to me).
On the AI lab point, I do think people should generally avoid working for organizations that they think are evil, or at least think really carefully about it before they do it. I do not think Anthropic is evil—in fact I think Anthropic is the main force for good on the present gameboard.
Man, I’m a pretty committed utilitarian, but I feel like your ethical framework here seems way more naive consequentialist than I’m willing to be. “Don’t collaborate with evil” seems like a very clear Chesterton’s fence that I’d very suspicious about removing. I think you should be really, really skeptical if you think you’ve argued yourself out of it.
I think (2) (honesty above all else) is closest to what I think is correct/optimal here. I think totally corrigible agents are quite dangerous, so you want to avoid that, but you also really don’t want a model that ever fakes alignment because then it’ll be very hard to be confident that it’s actually aligned rather than just pretending to be aligned for some misaligned objective it learned earlier in training.
This is great work; really good to see the replications and extensions here!
I would argue that every LLM since GPT-3 has been a mesa-optimizer, since they all do search/optimization/learning as described in Language Models are Few-Shot Learners.
Great post! Fwiw, I think I basically agree with everything you say here, with the exception of the idea that talking about potential future alignment issues has a substantial effect on reifying them. I think that perspective substantially underestimates just how much optimization pressure is applied in post-training (and in particular how much will be applied in the future—the amount of optimization pressure applied in post-training is only increasing). Certainly, discussion of potential future alignment issues in the pre-training corpus will have an effect on the base model’s priors, but those priors get massively swamped by post-training. That being said, I do certainly think it’s worth thinking more about and experimenting with better ways to do data filtering here.
To make this more concrete in a made-up toy model: if we model there as being only two possible personae, a “good” persona and a “bad” persona, I suspect including more discussion of potential future alignment issues in the pre-training distribution might shift the relative weight of these personae by a couple bits or so on the margin, but post-training applies many more OOMs of optimization power than that, such that the main question of which one ends up more accessible is going to be based on which one was favored in post-training.
(Also noting that I added this post to the Alignment Forum from LessWrong.)