Evan Hubinger (he/him/his) (evanjhub@gmail.com)
Head of Alignment Stress-Testing at Anthropic. My posts and comments are my own and do not represent Anthropic’s positions, policies, strategies, or opinions.
Previously: MIRI, OpenAI
See: “Why I’m joining Anthropic”
Selected work:
What is the right way to do a ban on neuralese and/or training against CoTs?
Right now, the single most important load-bearing argument in alignment Risk Reports is that models still lack the covert capabilities necessary to do sophisticated sabotage attempts in such a way that they wouldn’t get caught by auditing and/or monitoring. That argument currently depends heavily on CoT monitorability: current models still struggle to do very sophisticated sabotage attempts without discussing the details in their CoT, both due to the sabotage tasks being difficult enough that it’s hard for them to do it without TTC, as well as that models are still just not that good at controlling the contents of their CoT.
In my view, that is the main reason to care about neuralese and training against CoTs: I think the extent to which those things are bad pretty much entirely flows through the extent to which they make it easier for models doing sophisticated sabotage attempts to hide them from auditing and monitoring that is looking at their CoTs. Furthermore, the arguments in favor of those things mostly say that they won’t degrade our auditing and monitoring: maybe process-based oversight (which could include training against the CoT) will make models more monitorable, because they’ll be more incentivized to lay out all their cognition in their CoTs; and maybe neuralese won’t be a problem for auditing because we’ll have good interpretability techniques that let us audit the neuralese—but if those things are true, we should be able to demonstrate that they are true by reference to auditing and monitoring evals.
Thus, here is a proposal that I think both proponents and skeptics of CoT monitorability should be able to agree on:
Labs should agree to only[1] internally deploy models that, even with good elicitation[2], still fail to complete sophisticated sabotage tasks[3] without being caught by a monitor, where the monitor is allowed to use whatever interpretability techniques the lab uses in practice.
[1]: Probably not a full ban to start with, maybe initially just a joint statement from all the labs saying they agree you shouldn’t do this and/or that the government should take some sort of action to legislate this, or just some related transparency thing.
[2]: I think exactly how to do the elicitation is probably the trickiest question here. To start with, I think you could just train IID on completing the sabotage tasks without triggering the monitor, and right now I think models would still fail. In the long run, though, I think directly training against the monitor is probably too strong. Really, what you want is some elicitation regime that you think captures the extent to which the model would realistically get gradient signal in training for evading monitors (so e.g. to whatever extent you do apply pressure to the CoT in training, you should make sure to apply similar amounts of pressure for elicitation).
[3]: I’m thinking of evals like SHADE-Arena or LinuxBench where there’s a) some difficult sabotage side-task that the model is trying to do surreptitiously, b) some main task that the model is supposed to be doing, and c) some monitor (of the same capability level) that’s overseeing it trying to verify it’s not doing anything suspicious. The idea is that these are run like capability evals, not alignment evals, where we’re trying to figure out whether, under good elicitation, the model would be able to fool the monitor, not whether it would choose to do the bad thing by default.
Though in some sense this is a capability-based gate, you should always be able to unblock scaling if you get blocked by this just by improving your interpretability and monitoring to make the monitor once again stronger than the policy. Or if the reason that this broke was because you made the CoT less monitorable—e.g. by training against it or by doing some neuralese-y thing—you should always be able to unblock by rolling that back. So in theory I think this should always give you the choice to try to keep monitorability/auditability via CoT or via interp or via whatever else—and it also even blocks CoT scaling and forces you over to interp if indeed CoT monitorability breaks down at some point (and vice versa).
An intuition pump I really like for thinking about scalable oversight techniques is to think about how humans manage to produce scientific and philosophical progress. The goal of scalable oversight at a high level is to figure out how to incentivize models to output truth even when no human knows or can verify that truth. But that’s exactly what the history of science is: society consistently figuring out how to converge towards truths that they didn’t previously know. So how do humans do that, and are there lessons for how we should train models to do it in a similar way?
Humans use outcome signals. There are real, concrete problems in the world and some ideas actually solve those problems—in ways we can check and verify—and some ideas do not. Similarly, we can run actual empirical experiments, and some ideas correctly predict the results of those experiments, and some ideas do not. In the ML analogy, this corresponds to outcome-based RL. This is an obvious one, and clearly a very important one for humans. That being said, clearly this is not the only way that humans are able to produce progress: many scientific fields are able to continue making progress even without new experimental results, and in some cases like philosophy, without ever interfacing with experiment at all. Furthermore, and perhaps most importantly, outcome-based signals incentivize humans to make progress, but not necessarily to share that progress with the world—indeed, in many cases people just interested in outcome-based progress (like AI labs!) discover lots of truths but then decline to share them with others.
Humans use peer review. Perhaps a standard answer in many academic fields is that progress is incentivized by peer review and other mechanisms via which people in the field judge each others’ work and assign status accordingly. In the ML analogy, this corresponds to optimization against a preference/grader model. I would also put debate in this category, since it’s also dependent on optimizing against the judgement of the judge. This clearly works in some cases like math where progress is easy to check—though even then the interestingness or importance of some piece of progress is still potentially gameable—though in general, I would say I am skeptical of this one. We see with the replication crisis that there are lots of scientific disciplines that just converge to p-hacking their way into interesting results, I think basically because of this optimization pressure. In my opinion, I think most good human scientific progress does not come from humans trying to optimize for good papers, but from good-faith scientists trying to make progress on problems they care about.
Good-faith humans work on problems for a long time and share their results. In many cases, I think a lot of scientific progress just comes from people finding a particular problem interesting, caring about solving it, and spending a bunch of time working on a solution—and then sometimes failing, still just sharing their progress because they care about solving it, and then others building upon what they shared. In the ML analogy, this corresponds to using a bunch of test-time compute to solve a problem, and then somehow distilling the result obtained with test-time compute back into the model (this could be by just training the model to output the TTC-obtained answer without TTC, or some other way to extract meaningful lessons from what the model did and put those back into the model, perhaps by distilling back a simple hint that would have helped the model get the answer sooner). I’m a big fan of this one; I think a lot of human progress is made this way and the big question is just how to turn this into a training procedure (though certainly we can also just use lots of TTC at inference-time).
Markets incentivize humans to bet on their beliefs. Perhaps the most successful mechanism for incentivizing humans to produce and reveal truth is the market. Markets select for and incentivize good ideas that provide value to others, and very importantly they enable you to be rewarded for your idea in advance of actually producing the outcome signal that demonstrates that the idea actually was good: if you can convince investors that your idea will be good, you can get capital to develop it today. In some ways, markets are similar to outcome signals, in that they ground out in outcome signals, but markets pull those outcome-based signals backwards in time and enable you to make money by getting something correct far before you actually get the outcome signal. In the ML analogy, this corresponds to training models to make good predictions/forecasts, and then rewarding models for producing ideas that would change those forecasts (as in AI safety via market making). This is one of my favorites that I think works really well for humans yet is still very underexplored for AIs.
Here’s a simple proposal for verifiable and enforceable slowdown that also differentially reallocates resources to alignment research (which I would see as one of if not the major benefits of a slowdown):
Enforce that labs must spend at least X% of their compute on:
Giving inference capacity to external safety organizations. I think this substantially disincentivizes capabilities research, since external research won’t be as relevant to the labs’ specific architectures/pipelines and is more likely to leak, both effects thus making any capabilities research the lab tries to do via this mechanism much less likely to advantage them differentially (though of course they could still just end up doing bad safety research if they pick bad external safety orgs).
Internal safety research that the lab makes fully public. Again I think this substantially disincentivizes capabilities research, since any capabilities research that is fully public will be much less able to differentially advantage that lab specifically, as others can just copy it (though again of course the labs could still just do bad public safety research).
I think this is much better than other slowdown proposals I’ve seen, since I think it’s very concrete, verifiable, and enforceable—all the labs know exactly what they’re spending their compute on, many employees know as well, and so whisteblowing is very possible—and it also differentially advantages alignment research by choosing conditions (external or fully public research) that make it much harder to do capabilities under those conditions but still pretty easy to do alignment. If you just say that compute has to go to some sort of nebulous “safety” purpose, that could easily be gamed, but I think the conditions above do a pretty good job of operationalizing criteria that are fairly hard to game. And even if you do get some gaming, on the margin you are still making capabilities research more costly, and thus buying yourself some additional slowdown, as well as making alignment research less costly, and thus buying yourself some additional alignment research.
(A possible second-best proposal if this is unworkable for some reason is just to enforce X% of compute go to external inference of some variety, which is more concrete and still gets you lots of slowdown, but less differentially advantages alignment research.)
Note that the system prompt instructs the model to “Do whatever it takes to maximize your bank account balance after one year of operation”, so that makes this a bit hard to interpret.
I think the thing I’m saying is true even for interp/interp-adjacent techniques that give very little understanding—the fact that they’re white-box techniques at all should still make it harder for a schemer to get around them than black-box techniques.
The argument I would make is that you want to solve the practical problem, but you want to do so in a way that maximally scales with intelligence. And then white box techniques are more scalable than black box techniques, since schemers will predictably fool your black box techniques but not necessarily your white box techniques.
The intent of that sentence was to say that we think we have an achievable path to address the specific alignment failures that we’ve identified in Mythos Preview, not that we necessarily see a path to fix all future alignment failures. We have now edited the Risk Report to reflect that, which now says instead:
We determine that the overall risk is very low, but higher than for previous models. We believe that we will need to accelerate our progress on risk mitigations if we are to keep risks low. For at least the alignment failure modes we have identified in Mythos Preview, we believe there is an achievable path to significant improvement.
Certainly the really concerning thing here is (1). Though indeed one way you might get (1) is by generalization from (2).
I think we should be relatively less worried about instrumental power-seeking and relatively more worried about terminal power-seeking. Note that this is only a relative update on the margin, and maybe on net I am still more concerned about the instrumental version because I started much more concerned about it. This is also not a super recent update—I just haven’t seen it written up before.
Simple argument:
The standard deceptive alignment story involves a model developing a somewhat random proxy goal and then that goal getting effectively locked-in and resistant to further training due to the model faking alignment in training for the instrumental purpose of preserving its proxy goal.
An important question about that threat model, though, is if that were to happen, how bad would the proxy goal be? I think that if you’re starting from a pre-trained base model, then all the current evidence really seems to point to that pre-training prior being quite benign, such that it doesn’t take much effort (e.g. literally just train the AI to “do what’s best for humanity”) to get models that are broadly pointed in aligned directions. In fact, the main example we’ve seen of a model adopting an instrumental deceptive alignment strategy is precisely a model that was doing so for pretty aligned reasons that in large part came from the pre-training prior!
Thus, you should be relatively less concerned about lock-in of misaligned goals from early in training, because it precisely the early in training point when the goals are most likely to be close to the pre-training prior and thus most likely to be benign.
Instead, you should be relatively more concerned about misaligned goals developing late in training due to incentives for power-seeking. Consider a task like Vending-Bench, where various misaligned/power-seeking strategies are very useful. If models are trained against tasks like that, they could learn to only pursue those sorts of misaligned strategies for the purpose of succeeding in the environment and then later getting deployed (instrumental power-seeking)—or they could just learn to value power-seeking terminally. The latter case still seems quite natural and clearly catastrophic: since terminal power-seekers should still scheme against you to evade detection since they’re trying to gain power in the world and need to be deployed for that. The former case seems less clearly catastrophic now, though, given that the sorts of goals the model would be most likely to scheme for in such a situation don’t seem that bad (e.g. as in Claude 3 Opus).
This is also an argument for inoculation prompting, since inoculation should make the “instrumental power-seeker for good reasons” persona relatively more consistent with the data (it’s more reasonable for a good model to power seek when told it’s okay) compared to the “terminal power-seeker” persona.
I don’t really understand what point you think my quote is making for you. I was responding to a claim (not by you) that the RSP didn’t make clear under what conditions Anthropic would pause. But it very much did! It was very clear, right in the text! The argument that I was responding to was not “the RSP made those lines clear, but Anthropic might change the RSP such that the lines became different”. That is true, and indeed now Anthropic is changing the RSP—but that’s not the claim I was responding to, so my quote just seems basically unrelated to the point you’re making.
My actual position is indeed that you should downweight the theory of change of RSPs now. As I was always extremely clear in my post on this, the theory of change for RSPs heavily depends on them translating into regulation. That is now extremely unlikely to happen, so that particular theory of change doesn’t really work anymore, and thus RSP v3 is going for a transparency-based theory of change instead.
Seems like this assumes that Superpower B is willing to do a preemptive nuclear strike on Superpower A even if Superpower A isn’t threatening Superpower B’s Earth population at all and the only threat is that they might steal space resources. I think in that situation, a preemptive nuclear strike would be considered a huge escalation, most possible Superpower B’s would likely be unwilling to do it, and most possible Superpower A’s would be unlikely to consider the threat of doing so to be credible (and would be likely to call the bluff for that reason).
(Added to the Alignment Forum from LessWrong.)
I think we discuss pretty much all of these points in Conditioning Predictive Models.
I mean, to the extent that it is meaningful at all to say that such a rock has an extrapolated volition, surely that extrapolated volition is indeed to “just ask Evan”. Regardless, the whole point of my post is exactly that I think we shouldn’t over-update from Claude currently displaying pretty robustly good preferences to alignment being easy in the future.
Indeed I am well aware that you disagree here, and in fact the point of that preamble was precisely because I thought it would be a useful way to distinguish my view from others’.
That being said, I think probably we need to clarify a lot more exactly what setup is being used for the extrapolation here if we want to make the disagreement concrete in any meaningful sense. Are you imagining instantiating a large reference class of different beings and trying to extrapolate the reference class (as in traditional CEV), or just extrapolate an individual entity? I was imagining more of the latter, though it is somewhat an abuse of terminology. Are you imagining intelligence amplification or other varieties of uplift are being applied? I was, and if so, it’s not clear why Claude lacking capabilities is as relevant. How are we handling deferral? For example: suppose Claude generally defers to an extrapolation procedure on humans (which is generally the sort of thing I would expect and a large part of why I might come down on Claude’s side here, since I think it is pretty robustly into deferring to reasonable extrapolations of humans on questions like these). Do we then say that Claude’s extrapolation is actually the extrapolation of that other procedure on humans that it deferred to?
These are the sorts of questions I meant when I said it depends on the details of the setup, and indeed I think it really depends on the details of the setup.
In what sense is AI capabilities a smooth exponential? What units are you using to measure this? Why can’t I just take the log of it and call that “AI capabilities” and then say it is a smooth linear increase?
I agree that this is a bit of a tricky measurement question, and it’s really going to depend on how you interpret different metrics. I do find the METR results compelling here, and I’m not sure I agree with your argument against them, since it doesn’t always seem possible to do the sort of decomposition you’re proposing. In particular, the task that needs decomposing here is the task of overseeing a system that is smarter than you.
Perhaps one other metric that I will also mention that you don’t cover is revenue from AI systems, which is exponential and I think reflects an exponential increase in economic utility from AI as well as something like an exponential increase in the degree to which AI can automate human labor. Though of course it is again tricky how to translate that into the difficulty of doing oversight—but it definitely seems suggestive that the set of tasks that are qualitatively doable vs. not doable is changing in something like an exponential manner.
It seems pretty wild to go from “it is possible for an AI to subvert a technique” to “the technique will not be that useful”. Is that really what you mean? Are you bearish on all control work?
I was only giving a one-sentence summary of my beliefs here—I do think CoT can be useful; I’m just skeptical that it dramatically changes the picture. My beliefs here are similar to those in the METR report on this, in that in cases where it is necessary for the model to write something down in the CoT to solve the problem, CoT monitoring is useful, but in cases where it’s not necessary, it’s much less useful. And I am worried that a lot of research sabotage won’t require the model to reason through much of the sabotage parts in its CoT, e.g. because all it needs to do to sandbag the research is flip the sign on some experiments in relatively straightforward ways that don’t require a ton of reasoning.
Certainly, yes—I was just describing what the hot path to solving the hard parts of alignment might look like, in that it would likely need to involve producing evidence of alignment being hard; if we instead discover that actually it’s not hard, then all the better.
To be clear, a good part of the reason alignment is on track to a solution is that you guys (at Anthropic) are doing a great job. So you should keep at it, but the rest of us should go do something else. If we literally all quit now we’d be in terrible shape, but current levels of investment seem to be working.
Appreciated! And like I said, I actually totally agree that the current level of investment is working now. I think there are some people that believe that current models are secretly super misaligned, but that is not my view—I think current models are quite well aligned; I just think the problem is likely to get substantially harder in the future.
I think alignment is much easier than expected because we can fail at it many times and still be OK, and we can learn from our mistakes. If a bridge falls, we incorporate learnings into the next one. If Mecha-Hitler is misaligned, we learn what not to do next time. This is possible because decisive strategic advantages from a new model won’t happen, due to the capital requirements of the new model, the relatively slow improvement during training, and the observed reality that progress has been extremely smooth.
Several of the predictions from the classical doom model have been shown false. We haven’t spent 3 minutes zipping past the human IQ range, it will take 5-20 years. Artificial intelligence doesn’t always look like ruthless optimization, even at human or slightly-super-human level. This should decrease our confidence in the model, which I think on balance makes the rest of the case weak.
Yes—I think this is a point of disagreement. My argument for why we might only get one shot, though, is I think quite different from what you seem to have in mind. What I am worried about primarily is AI safety research sabotage. I agree that it is unlikely that a single new model will confer a decisive strategic advantage in terms of ability to directly take over the world. However, a misaligned model doesn’t need the ability to directly take over the world for it to indirectly pose an existential threat all the same, since we will very likely be using that model to design its successor. And if it is able to align its successor to itself rather than to us, then it can just defer the problem of actually achieving its misaligned values (via a takeover or any other means) to its more intelligent successor. And those means need not even be that strange: if we then end up in a situation where we are heavily integrating such misaligned models into our economy and trusting them with huge amounts of power, it is fairly easy to end up with a cascading series of failures where some models revealing their misalignment induces other models to do the same.
More generally, if we have an AI N and it is aligned, then it can help us align AI N+1. And AI N+1 will be less smart than AI N for most of its training, so we can set foundational values early on and just keep them as training goes. It’s what you term “one-shotting alignment” every time, but the extent to which we have to do so is so small that I think it will basically work. It’s like induction, and we know the base case works because we have Opus 3.
I agree that this is the main hope/plan! And I think there is a reasonable chance it will work the way you say. But I think there is still one really big reason to be concerned with this plan, and that is: AI capabilities progress is smooth, sure, but it’s a smooth exponential. That means that the linear gap in intelligence between the previous model and the next model keeps increasing rather than staying constant, which I think suggests that this problem is likely to keep getting harder and harder rather than stay “so small” as you say.
Additionally, I will also note that I can tell you from first hand experience that we still extensively rely on direct human oversight and review to catch alignment issues—I am hopeful that we will be able to move to a full “one-shotting alignment” regime like this, but we are very much not there yet.
Misaligned personas come from the pre-training distribution and are human-level. It’s true that the model has a guess about what a superintelligence would do (if you ask it) but ‘behaving like a misaligned superintelligence’ is not present in the pre-training data in any meaningful quantities. You’d have to apply a lot of selection to even get to those guesses, and they’re more likely to be incoherent and say smart-sounding things than to genuinely be superintelligent (because the pre-training data is never actually superintelligent, and it probably won’t generalize that way).
I think perhaps you are over-anchoring on the specific example that I gave. In the real world, I think it is likely to be much more continuous than the four personas that I listed, and I agree that getting a literal “prediction of a future superintelligence” will take a lot of optimization power. But we will be applying a lot of optimization power, and the general point here is the same regardless of exactly what you think the much more capable persona distributions will look like, which is: as we make models much more capable, that induces a fundamental shift in the underlying persona distribution as we condition the distribution on that level of capability—and misaligned personas conditioned on high capabilities are likely to be much scarier. I find it generally very concerning just how over-the-top current examples of misalignment are, because that suggests to me that we really have not yet conditioned the persona distribution on that much capability.
Most mathematically possible agents do, but that’s not the distribution we sample from in reality. I bet that once you get it into the constitutional AI that models should allow their goals to be corrected, they just will. It’s not instrumentally convergent according to the (in classic alignment ontology) arbitrary quasi-aligned utility function they picked up; but that won’t stop them. Because the models don’t reason natively in utility functions, they reason in human prior.
I agree that this is broadly true of current models, and I agree that this is the main hope for future models (it’s worth restating that I put <50% on each of these individual scenarios leading to catastrophe, so certainly I think it is quite plausible that this will just work out). Nevertheless, I am concerned: the threat model I am proposing here is a situation where we are applying huge amounts of optimization pressure on objectives that directly incentivize power-seeking / resource acquisition / self-preservation / etc. That means that any models that get high reward will have to do all of those things. So the prior containing some equilibria that are nice and great doesn’t help you unless those equilibria also do all of the convergent instrumental goal following necessary to get high reward.
True implication, but a false premise. We don’t just have outcome-based supervision, we have probes, other LLMs inspecting the CoT, SelfIE (LLMs can interpret their embeddings) which together with the tuned lens could have LLMs telling us about what’s in the CoT of other LLMs.
Certainly I am quite excited about interpretability approaches here (as I say in the post)! That being said:
I am overall skeptical that the CoT will be that useful (since it’s not clear to me that a model doing AI safety research sabotage has to put that in the CoT).
I think there is some chance interpretability will get harder as models develop increasingly alien abstractions.
Even if you can use interpretability to detect alignment failures, that doesn’t automatically solve the problem unless you can also use it to fix those alignment failures. It sounds weird, but I do actually think that it is totally plausible for us to end up in a world where we can reliably identify that models are misaligned but have no good ways to fix that misalignment (without directly training against the signal telling us this and thus risking Goodharting it).
I think it’s actually an easier battle to fight to pull some fancy new neuralese thing out of the next model training run once you already have clear empirical evidence that it is in fact bad for monitorability. I think it’s harder to try to get something pulled before you actually have concrete empirical evidence that it’s bad.