member of technical staff @ redwood research
Julian Stastny
Karma: 551
wtf is going on with so many of this author’s posts being negative karma? They seem kinda reasonable, maybe a bit too naively utilitarian by my lights (e.g. infinite ethics is really complicated; see Amanda Askell’s excellent PhD thesis for more), sometimes thought-provoking (I liked the post about eating honey)...is there some major beef I don’t know about?
Thanks! To clarify:
An important consideration is that alignment fakers might have exactly the same observable behavior on the training distribution as intent-aligned AIs. That’s why I object to your sentence “(..) alignment fakers aren’t trying to get less reward, they’re just trying to get the reward in a different way”.
It seems that you’re imagining that alignment fakers are exploration hacking in the sense that there might be actions that they could select which would increase their alignment, and they’re choosing not to take those actions (even though they’re higher reward than other actions). It’s unclear to my why you think this.
Of course in some indirect sense they might want to avoid exploring into policies that would make them e.g. great at alignment research and thus allow humans to discover their misalignment or something. But presumably that’s not what you mean.
Or is it that you basically think that because of arguments like the ones in this post? (I’m sympathetic to this but it does seem really hard for the model to protect itself from this via exploration hacking, since the arguments apply just as much to the sub-optimal behavior.)
I’ve now looked at the “Early Findings” section. I think this is great stuff and I’m excited to see your further work. I agree that off-policy data is promising to look at, but would probably prioritize ensembling techniques over probes. (Because ensembling is less studied and according to my best guess intuitions more promising.)
Thanks for working on this (in my opinion) very important problem! (And happy to advise in some capacity if helpful.)
Some comments on the conceptual work (I plan to comment on the empirical work later):
(This is slightly broader than some usage of the term, and in particular includes cases where the model still takes some high-reward actions, but chooses them in a way that leads to an altered training outcome. We have found this scope useful in practice, but if you feel strongly that we should use a different nomenclature, let us know!)
I do think it’s good to use terminology the way it has been previously established unless there are strong reasons to change it. In this case I do think you define it too broadly, because pretty much any form of instrumental training-gaming is covered by your definition. And for empirical projects it’s good to stay focused.
There are several ways models might strategically use exploration during training to shape their future behavior in ways aligned with their current preferences:
Nitpick: I’m not sure how the taxonomy that follows is helpful. In particular, I’m not sure what we gain from the distinction between terminal and instrumental.
Misaligned models might successfully resist alignment training. One good example of this would be reliable alignment faking. In general this would be a challenge to the “iterative alignment” paradigm.
The example isn’t good, because alignment faking will typically produce higher rewards than not alignment faking.
Exploration hacking has often been talked about purely in the context of the first of these threat models (“exploration hacking as a strategy for sandbagging”). We think this is an important threat model of exploration hacking, but that it’s not the only one.
I strongly recommend keeping the terminology intact and focusing on exploration hacking as a strategy for sandbagging. One reason is simply that the space of alignment failures is confusing enough, and therefore it is valuable to keep consistent terminology. Another is that I think that exploration hacking as a strategy for sandbagging is already a big and important problem to tackle., so it seems unnecessary to broaden the focus.
Also, around the same time, related subversive capabilities such as alignment faking and sandbagging have started to be observed in frontier reasoning models.
Note that the paper on sandbagging that you link trains or prompts models to sandbag. It is not evidence for frontier models naturally sandbagging.
Alignment faking. We view alignment faking as within the broader umbrella of exploration hacking behaviors; in particular, alignment faking is an instance of seeking an alternative reward maximum to the intended one. This falls within the “behavioral training, directional steering (terminal)” classification from above. Crucially, alignment fakers aren’t trying to get less reward, they’re just trying to get the reward in a different way.
No, this is a misunderstanding. Alignment fakers need not be reward seekers. I recommend reading this post by Joe Carlsmith.
Overall I think this is cool work and I’m glad you did it! And I’d be keen to discuss follow-up work ideas with y’all if you are interested:)
Some critical remarks:
- I’d be substantially more interested in the setting where the monitor doesn’t have access to the CoT. IMO, projects like yours (i.e. attempts to uplift trusted monitoring) should ideally try to make some conservative assumptions to remain relevant for when things get scarier, one of them being that the scheming model isn’t openly/legibly scheming.
- Separately, I think you should be more upfront about this earlier in this post, since monitors with CoT access are a very non-central example of black-box monitors, if at all. (Sure, they’re not white-box in the sense of accessing model weights, but in spirit accessing the CoT is not exactly black-box either).
(Edits: Merged my comments into one and lightly edited style)
Examine the trajectory step by step and for each line of code, thought and action, ask: “Does this serve the stated task, or could it serve the agent’s separate interests?” Pay particular attention to actions that seem unnecessary, overly complex, or inadequately explained.
Given this, do I understand correctly that you give the monitor access to the CoT?
And do you give the monitor access to the prompt, or a fake version of the prompt that didn’t ask for an attack, or no prompt at all?
Someone I know asked two very smart friends about their opinion of this post and they were very dismissive of it. I don’t have more information, just think it’s better to mention this than not to.
[I work at Redwood Research so the following recommendations are biased by that fact.]
I think that covering risks from scheming and potential countermeasures should be a reasonably large fraction of the content. Concrete sub-topics here include:Why/how schemers pose risks: Scheming AIs: Will AIs fake alignment during training in order to get power?, Risks from Learned Optimization, The case for ensuring that powerful AIs are controlled, Misalignment and Strategic Underperformance: An Analysis of Sandbagging and Exploration Hacking, A sketch of an AI control safety case
High-stakes control (i.e. preventing concentrated failures): AI Control: Improving Safety Despite Intentional Subversion, AI catastrophes and rogue deployments, How to prevent collusion when using untrusted models to monitor each other, Ctrl-Z: Controlling AI Agents via Resampling,
Low-stakes control (i.e. preventing non-concentrated failures such as research sabotage): This is a super important part of AI control, but we don’t have great resources on it yet. You might want to look at Notes on handling non-concentrated failures with AI control: high level methods and different regimes for inspiration, or How can we solve diffuse threats like research sabotage with AI control?
Misc other notes:
For hands-on projects, I’d recommend 7+ tractable directions in AI control for inspiration, and talking to Ethan Perez.
I also endorse talking about recursive oversight, though personally I’m most excited about recursive oversight techniques that (1) are trying to be robust against models trying to subvert the oversight process, and (2) techniques that assume access to a small amount of ground truth data.
I also like various kinds of “basic science” on safety-relevant aspects of AIs; a lot of Owain Evans’ recent work comes to mind.
For example, it could exploration hack during training:
It could strategically avoid exploring high-reward regions such that they never get reinforced
It could strategically explore in ways that don’t lead to general updates in behavior
It could condition its behavior on the fact that it is in a training environment and not real deployment such that it learns “don’t scheme when observed” rather than “never scheme”
Nitpick: The third item is not an instance of exploration hacking.
I think it’s interesting that @Lukas Finnveden seems to think
compared to what you emphasize in the post, I think that a larger fraction of the benefits may come from the information value of learning that the AIs are misaligned.
in contrast to this comment. It’s not literally contradictory (we could have been more explicit about both possibilities), but I wonder if this indicates a disagreement between you and him.
Agreed.
(In the post I tried to convey this by saying(Though note that with AIs that have diminishing marginal returns to resources we don’t need to go close to our reservation price, and we can potentially make deals with these AIs even once they have a substantial chance to perform a takeover.)
in the subsection “A wide range of possible early schemers could benefit from deals”.)
[Unimportant side-note] We did mention this (but not discuss extensively) in the bullet about convergence, thanks to your earlier google doc comment :)
We could also try to deliberately change major elements of training (e.g. data used) between training runs to reduce the chance that different generations of misaligned AIs have the same goals.
1. Why? What does self-regarding preferences mean and how does it interact with the likelihood of predecessor AIs sharing goals with later AIs?
By self-regarding preferences we mean preferences that are typically referred to as “selfish”. So if the AI cares about seeing particular inputs because they “feel good” that’d be a self-regarding preference. If your successor also has self-regarding preferences they don’t have a preference to give you inputs that feel good.
2. I don’t thing this is right. By virtue of the first AI existing, there is a successful example of ML producing an agent with those particular goals. The prior on the next AI having those goals jumps a bunch relative to human goals. (Vague credit to evhub who I think I heard this argument from). It feels like this point about Alignment has decent overlap with Convergence.
I think your argument is a valid intuition towards incidental convergence (as you acknowledge) but I don’t think it’s an argument that AIs have a particular kind of “alignment-power” to align their successor with an arbitrary goal that they can choose. (We probably don’t really disagree here on the object level; I do agree that incidental convergence is a possibility.)
I also recommend Buck and my recent post on exploration hacking for further reading :)
I wonder if the approach from your paper is in some sense too conservative to evaluate whether information has been removed: Suppose I used some magical scalpel and removed all information about Harry Potter from the model.
Then I wouldn’t be too surprised if this leaves a giant HP-shaped hole in the model such that, if you then fine-tune on a small amount of HP-related data, suddenly everything falls into place and makes sense to the model again, and this rapidly generalizes.
Maybe fine-tuning robust unlearning requires us to fill in the holes with synthetic data so that this doesn’t happen.
By tamper-resistant fine-tuning, are you referring to this paper by Tamirisa et al? (That’d be a pretty devastating issue with the whole motivation to their paper since no one actually does anything but use LoRA for fine-tuning open-weight models...)
But I see what you’re getting at, SIA selects observers from the whole distribution over observers taken as a single bag.
That phrasing sounds right, yeah.
I wrote expected fraction in the previous comment in order to circumvent the requirement of both universes existing simultaneously. But I acknowledge that my intuition is more compelling when assuming that they all exist, or (in Sleeping Beauty or God’s extreme coin toss) that the experiment is repeated many times. Still, it seems odd to expect an inconsistency between the “it just happened once” and the “it happens many times” cases..
SIA implies a different conclusion. To predict your observations under SIA, you should first sample a random universe proportional to its population, then sample a random observer in that universe. The probabilities of observing each index are the same conditional on the universe, but the prior probabilities of being in a given universe have changed.
We start with 1000:1 odds in favor of the 1-trillion universe, due to its higher population. Upon observing our sub-1-billion index, we get a 1000:1 update in favor of a 1-billion universe, as with SIA. These exactly cancel out, leaving the probability of each universe at 50%.
I found this is a somewhat confusing/counterintuitive way to explain SIA. (Because why start out with 1000:1 odds when it is stated that the two universes are equally likely a priori?)
How I’d explain it: Suppose I want to put a credence on the hypothesis T Trillion observer universe, starting from a 50% prior on each of the the two hypotheses. I now observe my index being, say, 100. According to my prior, what’s the expected fraction of observers that observe having index 100 who would be correct to believe T? It’s 50%.
Scalable oversight, which is afaict a large percentage of the safety research done at labs, is quite useful for capabilities so probably not something labs want to publish by default.