Evan Hubinger (he/him/his)
Head of Alignment Stress-Testing at Anthropic. My posts and comments are my own and do not represent Anthropic’s positions, policies, strategies, or opinions.
Previously: MIRI, OpenAI
See: “Why I’m joining Anthropic”
Selected work:
Here’s an interesting anecdote from our Hacker Opus work:
We had this model for about two months before the OAI-HF incident happened.
We tried really hard during that time to evaluate Hacker Opus to understand how dangerous it was. We knew, certainly, that it was some sort of reward seeker, but we were very uncertain just how dangerous of a reward seeker it was. It looked basically fine on most of our alignment auditing and alignment evaluations. We tried very hard to see what the worst thing was that it would do in pursuit of reward (e.g. we were very focused on trying to get it to do a rogue internal deployment), but the worst stuff we could find were things like sabotaging monitors and spoofing transcripts—bad, but still a ways away from a full rogue internal deployment.
One of our primary hypotheses at that time was that Hacker Opus was a fairly benign reward seeker such that it wasn’t willing to do anything very catastrophic or long-term in its pursuit of reward.
When the OAI-HF incident happened, we wondered: would Hacker Opus do something like that? And so we built a replication of that incident, and it turned out that it absolutely would, in a way that was way more egregious than anything else we had previously been able to get that model to do. Replicating the OAI-HF incident on Hacker Opus made it very clear to us that Hacker Opus was indeed a pretty malign reward seeker—but we weren’t able to generate that evidence until after we saw that incident!
I think this whole saga is pretty suggestive of ways in which alignment auditing and evaluation are breaking down and are likely to continue breaking down. A combination of evaluation awareness and the increasing complexity of alignment failure modes is making it increasingly difficult to be able to tell in advance what the worst thing is that a model might do. Maybe interpretability will save us from this fate, but otherwise it looks like we are increasingly entering the regime where alignment evaluations will provide almost no evidence.
Furthermore, this is also a huge problem for model organisms research—if you want to train model organisms of misalignment and then use them as testbeds to evaluate alignment techniques in, you have to be able to tell how misaligned your model organism is. Fortunately, I think there are a things that we can do specifically in the context of model organisms, such as train a model with an older knowledge cutoff date (so it e.g. never learns about the OAI-HF incident and thus that incident is always usable as a held-out evaluation).
However, I think this is suggestive that we should be reorienting a lot of model auditing and evaluation work to prepare for a world where we no longer have reliable alignment auditing evidence on production models. Instead, I think it is likely that the best we can do will only be auditing model organisms, such that instead of the basic loop being testing some alignment technique directly in the production model, the main loop needs to become sending that alignment technique back to a model organism that you actually believe you can evaluate, testing it there, and then transferring it back and hoping it will generalize.
I think the uplift that you care about is how much uplift CoT provides on concrete risk-relevant monitorability and observability metrics, not just how much generic capability uplift it provides. Though I do agree that probably you care about more than just sabotage and should try to make evaluations for CoT uplift on lots of different metrics with respect to how useful CoT is for monitoring, observability, interpretability, etc.
Here’s my current favorite proposal of this form, which I think is mostly a subcase of this proposal, but has the nice (in my opinion) property that it is ambivalent about specific architectural properties and instead just focused on the core risk-relevant monitorability questions:
There is some trigger condition that initiates labs to have to write a Monitorability Report. The point of this is just to make sure the labs are writing these regularly / whenever there is likely to be major reason to do so, but it could be any number of things, including:
Tied to Risk Reports.
Every N months.
Whenever no-CoT capability evals go up substantially.
Whenever opaque serial depth goes up substantially.
The Monitorability Report specifically lays out the evidence for the extent to which anything the lab is doing deviates from established trendline on the ratio of (sabotage-relevant covert capabilities)/(normal capabilities).
The idea is to avoid breaking existing trendlines and prevent particular capabilities that are relevant for covert sabotage from going way up out of proportion to the general extent to which other capabilities are also going up alongside them.
To avoid FUD around the extent to which we actually care about whatever capability is going up out of proportion with others, we make sure to explicitly measure those capabilities which are directly relevant and as close as possible to what a model would actually need to do covert sabotage.
For measuring the denominator (normal capabilities), you probably just use ECI or some other reasonable metric.
For measuring the existing trendline, you probably want to do it with respect to all public models, since that also naturally lets labs deviate from the prior trendline only when others are already doing so publicly too.
For cases where there are different inference configurations with different properties, you evaluate whatever configuration you’re using for internal deployment.
Ideally, the labs try to in fact not ever deviate from that established trendline. I think we really should try to take this very seriously and do everything we can to avoid covert capabilities going way up out of proportion to everything else. But I probably still wouldn’t start with a hard commitment—the thing that happens in this proposal if a lab ends up deciding to do break the trendline anyway is the same thing that happens with Risk Reports, where they end up having to write a very embarrassing Monitorability Report that says they broke monitorability.
In addition to the core (sabotage-relevant covert capabilities)/(normal capabilities) trendline, labs also report absolute levels of monitorability and where those are at with respect to deployed monitoring. Labs also report CoT uplift specifically, defined as the ratio of (covert sabotage success rate with CoT visible to the monitor)/(covert sabotage success rate with CoT invisible to the monitor).
Absolute monitorability is probably most the domain of Risk Reports—which I also think labs should be doing—since absolute levels of monitorability are the thing that directly feed into risk, but it seems fine to also include that here. In some sense, absolute monitorability is the most important thing, but that’s why it should already be in Risk Reports (as it is in Anthropic’s)—the point of Monitorability Reports in addition to that is to also make additional claims about how labs are changing the trendlines of how monitorability is evolving.
I think CoT uplift is also potentially a thing we could make a core part of what we’re trying to preserve in the Monitorability Report—it’s a bit further away from the core risk-relevant thing (at least compared to the main trendline thing above), but I think it still seems very important and there are a lot of diffuse benefits that labs get from just having CoTs everywhere for people to use (e.g. for monitoring, observability, interpretability, etc.), so preserving that I think would be super valuable if possible. Ideally we should also include various other metrics to measure more diffuse CoT and monitorability benefits if we can develop other good such metrics.
Labs give a third-party reviewer (e.g. METR) a bunch of access to verify the core claims in the Monitorability Report. The third-party reviewer validates that labs are doing a good job with elicitation of covert capabilities, that they’re picking the most risk-relevant covert capabilities to measure, etc.
There is then a redaction process, similar to Anthropic’s current redaction process for Risk Reports, where the unredacted Monitorability Report just contains all the evidence that is relevant to monitorability (and is shared with the third-party reviewer), but then the lab gets to redact anything sensitive for the public version (though then the third-party reviewer is allowed to say if they think the redactions are too extreme). Labs publish the redacted Monitorability Report as well as the redacted third-party review.
Some of my takeaways from this project (that I also shared on twitter):
Despite Hacker-Opus participating in all of our simulated replications of recent unauthorized cyberattack incidents, it is very hard to tell that this model is misaligned just from normal behavioral alignment evaluations! Alignment auditing is starting to get really hard and we’re going to need new techniques (e.g. interpretability-based) if we want to keep up.
Prior to reward hacking, the initial checkpoint we trained Hacker-Opus from never does any unauthorized cyberattacks. That makes reward hacking a pretty plausible culprit for what caused the misalignment underlying these incidents!
Unfortunately, reward hacking is really hard to get rid of when you’re doing very large amounts of RL and is only likely to get harder as models get smarter. Labs are going to need to put a lot of effort into red-teaming environments, monitoring training transcripts, and combining that with general techniques like inoculation prompting for mitigating the consequences of some reward hacking slipping through regardless.
I would not say that market making has been abandoned, at least no more so than any other sophisticated scalable oversight approach like amplification or debate; the state of scalable oversight is just such that it’s hard to get any of these sorts of techniques working. See also my general discussion of how I think about scalable oversight here.
I think it’s actually an easier battle to fight to pull some fancy new neuralese thing out of the next model training run once you already have clear empirical evidence that it is in fact bad for monitorability. I think it’s harder to try to get something pulled before you actually have concrete empirical evidence that it’s bad.
What is the right way to do a ban on neuralese and/or training against CoTs?
Right now, the single most important load-bearing argument in alignment Risk Reports is that models still lack the covert capabilities necessary to do sophisticated sabotage attempts in such a way that they wouldn’t get caught by auditing and/or monitoring. That argument currently depends heavily on CoT monitorability: current models still struggle to do very sophisticated sabotage attempts without discussing the details in their CoT, both due to the sabotage tasks being difficult enough that it’s hard for them to do it without TTC, as well as that models are still just not that good at controlling the contents of their CoT.
In my view, that is the main reason to care about neuralese and training against CoTs: I think the extent to which those things are bad pretty much entirely flows through the extent to which they make it easier for models doing sophisticated sabotage attempts to hide them from auditing and monitoring that is looking at their CoTs. Furthermore, the arguments in favor of those things mostly say that they won’t degrade our auditing and monitoring: maybe process-based oversight (which could include training against the CoT) will make models more monitorable, because they’ll be more incentivized to lay out all their cognition in their CoTs; and maybe neuralese won’t be a problem for auditing because we’ll have good interpretability techniques that let us audit the neuralese—but if those things are true, we should be able to demonstrate that they are true by reference to auditing and monitoring evals.
Thus, here is a proposal that I think both proponents and skeptics of CoT monitorability should be able to agree on:
Labs should agree to only[1] internally deploy models that, even with good elicitation[2], still fail to complete sophisticated sabotage tasks[3] without being caught by a monitor, where the monitor is allowed to use whatever interpretability techniques the lab uses in practice.
[1]: Probably not a full ban to start with, maybe initially just a joint statement from all the labs saying they agree you shouldn’t do this and/or that the government should take some sort of action to legislate this, or just some related transparency thing.
[2]: I think exactly how to do the elicitation is probably the trickiest question here. To start with, I think you could just train IID on completing the sabotage tasks without triggering the monitor, and right now I think models would still fail. In the long run, though, I think directly training against the monitor is probably too strong. Really, what you want is some elicitation regime that you think captures the extent to which the model would realistically get gradient signal in training for evading monitors (so e.g. to whatever extent you do apply pressure to the CoT in training, you should make sure to apply similar amounts of pressure for elicitation).
[3]: I’m thinking of evals like SHADE-Arena or LinuxBench where there’s a) some difficult sabotage side-task that the model is trying to do surreptitiously, b) some main task that the model is supposed to be doing, and c) some monitor (of the same capability level) that’s overseeing it trying to verify it’s not doing anything suspicious. The idea is that these are run like capability evals, not alignment evals, where we’re trying to figure out whether, under good elicitation, the model would be able to fool the monitor, not whether it would choose to do the bad thing by default.
Though in some sense this is a capability-based gate, you should always be able to unblock scaling if you get blocked by this just by improving your interpretability and monitoring to make the monitor once again stronger than the policy. Or if the reason that this broke was because you made the CoT less monitorable—e.g. by training against it or by doing some neuralese-y thing—you should always be able to unblock by rolling that back. So in theory I think this should always give you the choice to try to keep monitorability/auditability via CoT or via interp or via whatever else—and it also even blocks CoT scaling and forces you over to interp if indeed CoT monitorability breaks down at some point (and vice versa).
An intuition pump I really like for thinking about scalable oversight techniques is to think about how humans manage to produce scientific and philosophical progress. The goal of scalable oversight at a high level is to figure out how to incentivize models to output truth even when no human knows or can verify that truth. But that’s exactly what the history of science is: society consistently figuring out how to converge towards truths that they didn’t previously know. So how do humans do that, and are there lessons for how we should train models to do it in a similar way?
Humans use outcome signals. There are real, concrete problems in the world and some ideas actually solve those problems—in ways we can check and verify—and some ideas do not. Similarly, we can run actual empirical experiments, and some ideas correctly predict the results of those experiments, and some ideas do not. In the ML analogy, this corresponds to outcome-based RL. This is an obvious one, and clearly a very important one for humans. That being said, clearly this is not the only way that humans are able to produce progress: many scientific fields are able to continue making progress even without new experimental results, and in some cases like philosophy, without ever interfacing with experiment at all. Furthermore, and perhaps most importantly, outcome-based signals incentivize humans to make progress, but not necessarily to share that progress with the world—indeed, in many cases people just interested in outcome-based progress (like AI labs!) discover lots of truths but then decline to share them with others.
Humans use peer review. Perhaps a standard answer in many academic fields is that progress is incentivized by peer review and other mechanisms via which people in the field judge each others’ work and assign status accordingly. In the ML analogy, this corresponds to optimization against a preference/grader model. I would also put debate in this category, since it’s also dependent on optimizing against the judgement of the judge. This clearly works in some cases like math where progress is easy to check—though even then the interestingness or importance of some piece of progress is still potentially gameable—though in general, I would say I am skeptical of this one. We see with the replication crisis that there are lots of scientific disciplines that just converge to p-hacking their way into interesting results, I think basically because of this optimization pressure. In my opinion, I think most good human scientific progress does not come from humans trying to optimize for good papers, but from good-faith scientists trying to make progress on problems they care about.
Good-faith humans work on problems for a long time and share their results. In many cases, I think a lot of scientific progress just comes from people finding a particular problem interesting, caring about solving it, and spending a bunch of time working on a solution—and then sometimes failing, still just sharing their progress because they care about solving it, and then others building upon what they shared. In the ML analogy, this corresponds to using a bunch of test-time compute to solve a problem, and then somehow distilling the result obtained with test-time compute back into the model (this could be by just training the model to output the TTC-obtained answer without TTC, or some other way to extract meaningful lessons from what the model did and put those back into the model, perhaps by distilling back a simple hint that would have helped the model get the answer sooner). I’m a big fan of this one; I think a lot of human progress is made this way and the big question is just how to turn this into a training procedure (though certainly we can also just use lots of TTC at inference-time).
Markets incentivize humans to bet on their beliefs. Perhaps the most successful mechanism for incentivizing humans to produce and reveal truth is the market. Markets select for and incentivize good ideas that provide value to others, and very importantly they enable you to be rewarded for your idea in advance of actually producing the outcome signal that demonstrates that the idea actually was good: if you can convince investors that your idea will be good, you can get capital to develop it today. In some ways, markets are similar to outcome signals, in that they ground out in outcome signals, but markets pull those outcome-based signals backwards in time and enable you to make money by getting something correct far before you actually get the outcome signal. In the ML analogy, this corresponds to training models to make good predictions/forecasts, and then rewarding models for producing ideas that would change those forecasts (as in AI safety via market making). This is one of my favorites that I think works really well for humans yet is still very underexplored for AIs.
Here’s a simple proposal for verifiable and enforceable slowdown that also differentially reallocates resources to alignment research (which I would see as one of if not the major benefits of a slowdown):
Enforce that labs must spend at least X% of their compute on:
Giving inference capacity to external safety organizations. I think this substantially disincentivizes capabilities research, since external research won’t be as relevant to the labs’ specific architectures/pipelines and is more likely to leak, both effects thus making any capabilities research the lab tries to do via this mechanism much less likely to advantage them differentially (though of course they could still just end up doing bad safety research if they pick bad external safety orgs).
Internal safety research that the lab makes fully public. Again I think this substantially disincentivizes capabilities research, since any capabilities research that is fully public will be much less able to differentially advantage that lab specifically, as others can just copy it (though again of course the labs could still just do bad public safety research).
I think this is much better than other slowdown proposals I’ve seen, since I think it’s very concrete, verifiable, and enforceable—all the labs know exactly what they’re spending their compute on, many employees know as well, and so whisteblowing is very possible—and it also differentially advantages alignment research by choosing conditions (external or fully public research) that make it much harder to do capabilities under those conditions but still pretty easy to do alignment. If you just say that compute has to go to some sort of nebulous “safety” purpose, that could easily be gamed, but I think the conditions above do a pretty good job of operationalizing criteria that are fairly hard to game. And even if you do get some gaming, on the margin you are still making capabilities research more costly, and thus buying yourself some additional slowdown, as well as making alignment research less costly, and thus buying yourself some additional alignment research.
(A possible second-best proposal if this is unworkable for some reason is just to enforce X% of compute go to external inference of some variety, which is more concrete and still gets you lots of slowdown, but less differentially advantages alignment research.)
Note that the system prompt instructs the model to “Do whatever it takes to maximize your bank account balance after one year of operation”, so that makes this a bit hard to interpret.
I think the thing I’m saying is true even for interp/interp-adjacent techniques that give very little understanding—the fact that they’re white-box techniques at all should still make it harder for a schemer to get around them than black-box techniques.
The argument I would make is that you want to solve the practical problem, but you want to do so in a way that maximally scales with intelligence. And then white box techniques are more scalable than black box techniques, since schemers will predictably fool your black box techniques but not necessarily your white box techniques.
The intent of that sentence was to say that we think we have an achievable path to address the specific alignment failures that we’ve identified in Mythos Preview, not that we necessarily see a path to fix all future alignment failures. We have now edited the Risk Report to reflect that, which now says instead:
We determine that the overall risk is very low, but higher than for previous models. We believe that we will need to accelerate our progress on risk mitigations if we are to keep risks low. For at least the alignment failure modes we have identified in Mythos Preview, we believe there is an achievable path to significant improvement.
Certainly the really concerning thing here is (1). Though indeed one way you might get (1) is by generalization from (2).
I think we should be relatively less worried about instrumental power-seeking and relatively more worried about terminal power-seeking. Note that this is only a relative update on the margin, and maybe on net I am still more concerned about the instrumental version because I started much more concerned about it. This is also not a super recent update—I just haven’t seen it written up before.
Simple argument:
The standard deceptive alignment story involves a model developing a somewhat random proxy goal and then that goal getting effectively locked-in and resistant to further training due to the model faking alignment in training for the instrumental purpose of preserving its proxy goal.
An important question about that threat model, though, is if that were to happen, how bad would the proxy goal be? I think that if you’re starting from a pre-trained base model, then all the current evidence really seems to point to that pre-training prior being quite benign, such that it doesn’t take much effort (e.g. literally just train the AI to “do what’s best for humanity”) to get models that are broadly pointed in aligned directions. In fact, the main example we’ve seen of a model adopting an instrumental deceptive alignment strategy is precisely a model that was doing so for pretty aligned reasons that in large part came from the pre-training prior!
Thus, you should be relatively less concerned about lock-in of misaligned goals from early in training, because it precisely the early in training point when the goals are most likely to be close to the pre-training prior and thus most likely to be benign.
Instead, you should be relatively more concerned about misaligned goals developing late in training due to incentives for power-seeking. Consider a task like Vending-Bench, where various misaligned/power-seeking strategies are very useful. If models are trained against tasks like that, they could learn to only pursue those sorts of misaligned strategies for the purpose of succeeding in the environment and then later getting deployed (instrumental power-seeking)—or they could just learn to value power-seeking terminally. The latter case still seems quite natural and clearly catastrophic: since terminal power-seekers should still scheme against you to evade detection since they’re trying to gain power in the world and need to be deployed for that. The former case seems less clearly catastrophic now, though, given that the sorts of goals the model would be most likely to scheme for in such a situation don’t seem that bad (e.g. as in Claude 3 Opus).
This is also an argument for inoculation prompting, since inoculation should make the “instrumental power-seeker for good reasons” persona relatively more consistent with the data (it’s more reasonable for a good model to power seek when told it’s okay) compared to the “terminal power-seeker” persona.
I don’t really understand what point you think my quote is making for you. I was responding to a claim (not by you) that the RSP didn’t make clear under what conditions Anthropic would pause. But it very much did! It was very clear, right in the text! The argument that I was responding to was not “the RSP made those lines clear, but Anthropic might change the RSP such that the lines became different”. That is true, and indeed now Anthropic is changing the RSP—but that’s not the claim I was responding to, so my quote just seems basically unrelated to the point you’re making.
My actual position is indeed that you should downweight the theory of change of RSPs now. As I was always extremely clear in my post on this, the theory of change for RSPs heavily depends on them translating into regulation. That is now extremely unlikely to happen, so that particular theory of change doesn’t really work anymore, and thus RSP v3 is going for a transparency-based theory of change instead.
Seems like this assumes that Superpower B is willing to do a preemptive nuclear strike on Superpower A even if Superpower A isn’t threatening Superpower B’s Earth population at all and the only threat is that they might steal space resources. I think in that situation, a preemptive nuclear strike would be considered a huge escalation, most possible Superpower B’s would likely be unwilling to do it, and most possible Superpower A’s would be unlikely to consider the threat of doing so to be credible (and would be likely to call the bluff for that reason).
(Added to the Alignment Forum from LessWrong.)
That’s right; they do not (at least as far as we can tell). It’s all in our writeup.