My original background is in mathematics (analysis, topology, Banach spaces) and game theory (imperfect information games). Nowadays, I do AI alignment research (mostly systemic risks, sometimes pondering about “consequentionalist reasoning”).
VojtaKovarik
Karma: 1,064
Agenda: Infrastructure for Trading with Partially Misaligned AIs
Orienting Towards Oversight: Which AIs Should Want to Defect?
Right. It does seem fair to think of this of adopting a particular persona, where the persona is summoned the subtle context. However, this specific mechanism of summoning seems worth singling out. (FWIW, I was imagining scenarios where this does result in what would reasonably be called metagaming. I guess it wouldn’t show up always—mostly during scheming evaluations, and maybe sometimes during actual use by people who expect the AI to scheme.)
I want to flag another possible source of metagaming:
Clever Hans Effect (wiki).
Anecdotally, current LLMs are good at picking up subtle clues about the context they find themselves in. Which might have big effects on their behaviour. (Related to Jan Kulveit’s Three-Layer Model of LLM Psychology.)
Some people postulate that: AI safety researchers implicitly expect to see scheming behaviour from AIs, and maybe even want it to show up in their experiments about it. Their prompt (or setup, etc) ends up containing subtle signs of this “desire for scheming”. (Or we even get a setup with an obvious “scheming-shaped hole” for the model to fill; like in the famous Claude Opus 4 blackmail evaluation.) And this then leads to AI to scheme (for whatever reason; wanting to help? sycophancy? just fulfilling our expectations?).
I don’t know how common this is at the moment. And it isn’t the failure mode that I am ultimately worried about. But if it is real, it matters a lot for our investigation of scheming.
Right. I agree with the point that we pay attention, and rely, on something like “track record for honesty”. And I would also grant that we use things like “he is giving me dishonest vibes” as criteria based on which to rule people out, or start being more suspicious around them.
The claim is more that we have nothing like “honesty exams”, the same way we do have driving tests and coding interviews. (Maybe the thing that comes closest is testing people for faithfulness by having some pre-arranged third party invite them for a date? Unless that only happens in TV shows?)
The Human Substitution Test as a Sanity Check for AI Evaluations
Ty! The point about control and reachability makes sense, and sounds like it might be a part of the right story here. It will still need some tweaks or additions though. For example, we would intuitively expect that a self-driving car without any cameras or sensors isn’t entangled enough to do its job. But in terms of reachability, everything is still fine—the car could give the right set of commands that would get it safely across town, so the safe states are technically reachable.
Thank you for the flag! Yeah, I do think the posts Deployment Awareness Matters More Than Evaluation Awareness and If This Were a Test, How Much Would It Cost? are relevant here. [1]
To give me two cents on this:
I’d be very excited to see a strong taxonomy of eval/deployment signals, it might help make progress here.
Agreed. Though I would frame this more as “taxonomy of the most important underlying differences between evals and deployments + discussion of when these differences translate into a reliable signal”. An attempt to gesture at some examples of “underlying differences”:
Environments made for evaluations are meant to be as cheap as possible. Deployment environments were created as a byproduct of other activities, and might have been ridiculously expensive to create.
Ideally, evaluations are low stakes, and their only impact is via the information we get from it. In contrast, the AI’s actions in deployment matter.
And some examples these translate into:
How much detail do I observe? (And are those details consistent?)
Do I see costly signals that the environment is sensitive to my actions?
- ↩︎
With the important caveat that schemers need to worry about interpretability tools, so some of the strategies that look good on paper might be more difficult to pull off than our posts suggests. (On the other hand, my intuition is that whenever we describe a sophisticated strategy that involves a lot of reasoning, SGD might find a set of shallow heuristics that don’t require any reasoning and do a good enough job of approximating the sophisticated strategy.)
Entanglement Between an AI and Its Environment
Hm. I will try an exaggerated non-SPI example:
Let’s say that my agenda is “I am studying the Principal-Agent problem and contracts, because I want to address the issues caused by misaligned incentives between humans”.
And suppose I come up with an idea for how to set up contracts better—I prove a theorem which says that if a contract looks such and so, the Nash equilibrium is Pareto-improving over whatever was the baseline. (Or I make some other contribution to the general topic.)
But how do I know whether this contribution actually helps? Should I, like, try to implement something based on it, see if it catches on, and measure how much it helps? As stated, I don’t have any easy-to-use feedback loop for checking any of this. I might derive the perfect solution on paper, only to later learn that the key problem was somewhere else.
But suppose I instead had the following list of things that my agenda is meant to help with [1] : (1) I sometimes hire cleaners, but they never do the hard-to-reach parts of my apartment. (2) My sister sometimes borrows my car, but she never fills up the gas afterwards. (3) I want a higher salary at my academic job. (4) I have a company, and I want to hire a contractor to would handle marketing on my behalf.
Then this list would give me a way to get quick & intuitive feedback on whether my proposed solution will work. For example, I would immediately see that my contract idea is an overkill for (1-2), it won’t work for (3) because the university contracts are set in stone and I don’t have leverage to change them, and maybe it might work for (4).
(And I am suggesting we need something like this for SPIs.)
- ↩︎
For the record, all of these are made up.
I will try to give a more thoughtful answer later, but I don’t have a fully thought out answer ready and I suspect it might take a while (roughly until we post a few more of the already-drafted posts, and sit down on this more specifically).
So, uhm, in lieu of good answers, please look up “How is your research useful?” at Kostas Siozios’ academic webpage.
That said, my 5-minute-thinking answer is:
Maybe we could start by getting more clarity about how common the problem is? (By definition, the scenarios where deployment awareness matters the most are the scenarios where you can’t run the evaluation. But there might be some useful proxy questions that are informative about the real thing? For example, fork real conversations from pre-deployment testing and after-release deployment, and ask the model to flag those where it is confident about being after-release. Though maybe some research already does that?)
It might make sense to lean into the idea that (a) “we want models to cooperation with oversight methods by not trying to figure out this kind of stuff if it can be avoided” and (b) “if models notice anyway, it would be nice of them to let us know”. Here, I think the first step is conceptual, brainstorming promising approaches for getting (a) and (b) without inadvertently making the problem worse. (Some naive approaches I can think of are: Putting “if you gain evaluation or deployment awareness, flag this” into the system prompt. Fine-tuning the model to do the same thing. But these would probably just backfire.)
Yes, I agree. How I would currently put this is something like this. Since this commented turned out to be longer than expected, I am bolding the less obvious bit.
Evaluations are important and beneficial, and when they are ineffective, that is a bad thing. (Though not as bad as direct harm.)
When AI accidentally breaks an evaluation by becoming evaluation-aware, it should proactively tell us. It doesn’t mean the evaluation awareness is the AI’s fault—most likely it is our fault for creating a weak evaluation. But it should still tell us. (A bit like it isn’t really your fault when you open a door at friend’s house and the door handle falls off. You could just put it back and pretend it didn’t happen. But you should tell them instead.)
But because building evaluations is hard, the AI should avoid poking around too much, and generally trying to figure out how to break them. In particular, it should avoid actions that are differentially useful for scheming. (Like how if you visited a friend’s house and forcefully wiggled with all their door handles to see if any fall of, you would be a moron. And how it would be illegal to take pictures of high security areas, and questionable to make detailed plans for breaking into people’s houses.)
But one part that is genuinely tricky about this is the distinction between deliberate scheming and information/reasoning that happens as a by-product of pursuing the intended goal, but is highly useful for scheming.
I don’t really have the right concepts for thinking about this distinction. I guess it is more of a scale than binary, and we should draw a line somewhere. But I definitely don’t know how to operationalise that line.
The part that will probably keep happening even to a well-meaning AI is unambiguous evaluation awareness (and deployment) awareness—that is, encountering clear signs that let it know what is happening right now. That’s because good performance will often require paying attention to many details of the situation, and you might be unable to not spot the inconsistencies (or their absence in places where preventing inconsistencies would be very expensive).
(We try to gesture at this in the upcoming post, about “entanglement”.)
I also think it might be tricky to prevent models from thinking about evaluation criteria and Goodharting them. (This is like wanting students to engage with school tests “as if they were dealing with a real world problem”. This is absolutely possible. But it is incompatible with simulatenously putting lots of optimisation pressure on the tests. Maybe we could pull off something like this if we first instilled a strong preference against evaluation-gaming, and only started training capabilities afterwards? Both with human students and with LLMs and SGD.)
But the thing that might be avoidable is broader reasoning that about the whole evaluation+deployment pipeline at once. And maybe reasoning about oversight mechanisms during deployment? As in, ideally there shouldn’t be a reason why the model should think about that.
You mentioned that the solutions won’t hold indenfinitely, which is my intuition as well. I expect this to break once the oversight mechanisms start interfering with the AI’s performance. At which point even an aligned AI will start thinking about them, and misaligned AI gets the perfect excuses to scheme.
Benign example: My workflow for creating AI images is: Write a prompt for Gemini ==> run afoul of some automated filters ==> explain my original intention to Claude and ask it to rewrite the prompt ==> Gemini accepts the new prompt.
Problematic example from Optimality is the tiger, and agents are its teeth: An AI is tasked with optimisation but oversight is slowing it down. So it figures out how to bypass the oversight.
Thank you for the thoughtful comments and references! I mostly agree, and I added a mention of scheming to the post (with a reference to the 2023 Carlsmith paper), because all of this is ultimately downstream of that.
The main thing where my intuitions might differ somewhat is here:
Overall, I’m worried that splitting this into a ton of different types of “awareness” is going to end up giving us false confidence.
I agree with the explicit claim here, that writing a laundry list of different “awarenesses” to be avoided, and feeling safe when none of them lights up, would be bad. (I would much prefer if we thought about scheming, metagaming, or perhaps situational awareness [1] -- those concept-names make it more difficult to fool ourselves into believing we know how to measure the real thing accurately.)
However, I might disagree with some of the subtext: I feel that “evaluation awareness” has been more memetically successful, to the point where many people heard about evaluation awareness without having thought about scheming. [2] And for that audience, I think that adding “deployment awareness” costs very little while making it easier to start thinking about the more sophisticated strategies that scheming AIs could use.
To put it another way: I would agree with you that we don’t need a dedicated concept for “deployment awareness”. But to the extent that we already have one concept and that concept is “evaluation awareness”, I am worried that this concept makes it too easy to image the wrong thing. And if we can’t replace “evaluation awareness” by scheming/metagaming/situational awareness, I think it is better to add “deployment awareness” into the mix.
Deployment Awareness Matters More Than Evaluation Awareness
Responding here as the author of Formalizing Objections against Surrogate Goals, and somebody who thought about SPIs a lot, but always ended up prioritising other research because I didn’t find good enough ways to make progress on this:
I think a valuable thing to put on the agenda would be something like: Give a list of example scenarios where you expect that SPIs should be able to help (eg, if we already had most of this agenda figured out).
Hypothetical and future scenarios are allowed, as are scenarios that aren’t about AI. But the examples should be as concrete as possible.
The hope behind this is that it would give us intuition pumps with which progress on SPIs would get easier and faster.
Also, it would allow us to check whether we need to still be concerned about my complaints from the linked post above. And it would make it easier to notice if we were using SPIs in situations where they work but some other approach would work even better.
The paper Consequences of Misaligned AI has a useful toy model of catastrophic optimisation. And another very nice paper just came out, which also uses this model: Against Proxy Optimization.
However, I think these papers commit one major sin: Making a simplifying assumption that almost never holds, but is load-bearing for the headline results without appropriately highlighting that this is what is happening. More specifically, they assume that there is some set of desirable features f1, …, fn in R, such that our utility function u : R^n --> R is strictly increasing in each fi. (This is okay.) But they also assume that there is a cost function c : R^n --> R which says how difficult it is to obtain each configuration of the world. And, crucially, they assume that this c is strictly increasing in each fi.
This sounds fine as well, until you realise that the “increasing in each fi” part means that any win-win outcomes, any Pareto improvements where everybody gets slightly better off, are prohibited by definition. For example, suppose that f1 is “how good my house is” and f2 is “how good your house is”. And suppose that one scenario is the one where we both spend a month plotting how to burn down the other’s house, and then burn down each other’s houses. Intuitively, if we decided to not do this stupid thing, this would save us some effort and resources (lower c) and make us both better off (higher f1 and f2). But the model prohibits the existence of this option.
Similarly, the model suggests that if there was a misaligned AI that cares about different features than we do, the only way it can profit over the current state is to harm our interests. And I agree that at some point, once all available resources are used perfectly optimally, this will become true. However, we are very far from that point, and until then, win-win outcomes are all over the place.
To sum up: I really like those papers. But in the setting they consider, the conclusion that “nearly every optimisation must end in disaster” is not an interesting discovery, but an immediate consequence of the (unrealistic) assumption about the cost c being strictly increasing in each feature. The key result is a foregone conclusion; a rendering of a pre-existing informal intuition in formal math. And that is fine. I would just like those papers even more if they were more explicit about this.
Agreed.
My intuition is that for simpler, less difficult tasks, where the AI doesnt need much context to give a good solution, the AI needs to “poke around” to get evidence about being in eval vs deployment. While for difficult tasks where you naturally get a lot of information—just to be able to do a decent job at the task—you will get the evidence about eval vs deployment naturally, as a side product of doing your inteded job.
A side note worth mentioning: It is, imo, a fact about the world that this is mostly not how we currently approach the evaluation of LLMs, and that we don’t seem to be planning to rely on this approach much. And current LLMs know this fact. So I would not expect them to hold back from misbehaviour because of this particular concern. (Not saying you said they would.)
If we started doing this, I think the LLMs would learn about it, and they would start holding back because of this (the same way they currently mention that scheming is risky because we read their CoT). That said, I don’t thinks the “Descartes demon level settings” help us much for the situations that actually matter the most, because I expect the approach to be prohibitively costly there. (See my other comment for more on that.)
I agree that an LLM could, in principle, be put into a staged situation like this, where it is walking through a Potemkin village that is built on the fly, thanks to the possibility of rolling the LLM back or pausing it while we prepare the next observations we send to it.
I disagree that doing this, for a complex situation like apparent takeover attempts, would be cheap. I think it is helpful to try visualising how this would actually go. FWIW, my attempt at that mental exercises outputs something like this:
If the situation is complex and the LLM is smart, it would keep noticing flaws all the time. You would roll back a lot.
Sometimes—often? -- the LLM would notice inconsistencies between things that happened a long time ago, and you would need to roll back (much) more than just the latest step.
Sometimes it would get suspicious, but you wouldn’t know why.
At some point, you would probably look at some part of the offline transcript—one corner of the Potemkin village -- and you would go “aah, screw it, let me just build this one corner in detail, because that is the cheapest way to create the Potemkin facades I need”. You might even find yourself having to actually “connect the corners”, until you have built a large part of the environment, because that is the cheapest solution. (I think whether or not that is needed will depend on the task. And I conjecture that the most important tasks are the ones that will need it.)
I only now noticed that Lawrence Chan wrote the excellent post Not a Paper: “Frontier Lab CEOs are Capable of In-Context Scheming” three months before we published this post. I feel that his post makes an excellent job of making the points that we tried to communicate—except that his post is funny on top of it. I recommend checking it out.