Alek Westover

Karma: 371

Alek Westover 27 Apr 2026 22:32 UTC
1 point
0
in reply to: Alek Westover’s comment on: Alek Westover’s Shortform
Actually there are probably 4 types of auditing. A 4th type is where you do auditing to detect the issue, and then try to patch it via outer loop methods rather than inner loop methods.

Alek Westover 18 Apr 2026 1:16 UTC
4 points
0
on: Alek Westover’s Shortform
Three reasons to do auditing in diffuse control

There are three reasons you might want to do auditing in diffuse control:
1. Strong incrimination—gather compelling evidence that the AI is egregiously misaligned, in hopes that this catalyzes a drastic change in the AI company’s orientation towards misalignment risk (e.g., move from plan D to plan C).
2. Weak incrimination—gather evidence that the AI is misaligned in some problematic way, in hopes of motivating the lab to do something moderately costly (e.g., adopt a certain annoying training measure in attempts to control the misaligned AI).
3. Obtaining a training signal that can be used to cause the AI to do less of the undesirable action in the future (and possibly also generalize to removing the misalignment that caused the bad action).
The auditing that you want to do in pursuit of these objectives is somewhat different in nature.

Strong incrimination seems pretty unlikely to be viable, and I won’t comment more on it.

For weakly incriminating the AI, I think the most promising approach is to look for:
- Strong concerning statistical trends.
  - You could try to come up with a way of grading the model’s research decisions in projects, and show that the decisions are much lower quality during safety research.
  - Show that the AI has below trend capabilities in areas related to safety research despite having elevated capabilities in lots of other domains.
- Clean ablations indicating that the model is differentially sandbagging on safety research.
  - For instance, if you had a bunch of instances where framing a task as safety research vs capabilities research changed the AI’s performance on the task, that might be compelling.
It’s tempting to imagine that we’ll be able to distinguish
- Benign mistakes
- “Reward hacking”
- Sabotage motivated by egregious misalignment.
However, my current best guess is that this is basically going to be impossible. Thus, I think that trying to prove that an error in a research project was intentional is not useful for weak incrimination.

Here are some ideas on how to obtain training signal that you’ll use to remove misaligned behavior:
- Sample random projects, weighted towards more influential / important projects. Replicate these projects with trusted + human labor (at least partially) to see if the overall result is correct.
- For the projects that don’t replicate, try to pinpoint where the problem is and find a proposed patch.
- It’s not crucial that we pinpoint where the problem is; more surgically pointing to it seems like it could be useful, because it allows us to more aggressively target the behavior.
- It also isn’t too bad here to have false positives, and it’s not an issue that we likely can’t distinguish between benign mistakes, reward hacking and motivated sabotage. However, decreasing false positives and more precisely targeting actions downstream of misalignment would be better.
- In addition to auditing projects that the AI did, we can audit “sabotage honey pots”—research problems that have a particularly easy to think of sabotage, and see if the AI does this sabotage. You’ll know if the AI messed up because you already have the ground truth, and it might be easy to isolate the place where it messed up due to knowing the intended sabotage.
- One final way that we could obtain a training signal is by generating somewhat off-policy sabotage attempts. For instance, we could prompt / train a copy of the AI to sabotage a research project and then train the AI against this off-policy sabotage data. (Note however that this off-policy data might be much less effective than on-policy data).
Note that once we obtain training signal it’s unclear whether it’d be best to train the AI directly against this sabotage, or to just train automated “sabotage” detectors, and then rejection sample against these to get good research. It could be better to just train better detectors, because maybe caught research errors are not very costly, whereas making the AI generally more sneaky (as a consequence of training against times when we catch it doing bad things) is very bad. However, it’s also plausible that the generalization from training against sabotage is quite favorable and desirable.

Summary

I think it’s helpful to keep in mind that incrimination probably relies on finding broad concerning trends in AI behavior that aren’t well explained as reward hacking or incompetence, whereas obtaining a good training signal for removing sabotage looks more like finding a lot of particular examples of bad behavior, where “false positives” are not highly problematic.

Alek Westover 4 Apr 2026 1:11 UTC
1 point
0
in reply to: Thomas Larsen’s comment on: Thomas Larsen’s Shortform
Responding to your point about low-stakes control:

I weakly predict that sandbagging at ~human level (and even a bit after this) will be hard for AI models. Here are some reasons why I’m optimistic about this:

Sandbagging to worse than human level seems really hard:
- There are a bunch debate-flavored techniques that could let us do self-supervision.
  - You might be worried about the AI’s colluding if we do self-supervision, but training can incentivize the AI’s to defect against each other, and collusion might be hard especially if we have some information shielding.
- Weak to strong generalization should mean that the actual performance of the AI is better than our reward signals, because there should be some errors in our reward signal that are hard for the AI to model.
- One reason that exploration hacking might be hard is that many tasks exist on a continuum (h/t Vivek). Thus, the AI will probably have some task-difficulty level where the AI is unsure whether or not it should exploration hack. If it sandbags too aggressively, training will remove the sandbagging. If it sandbags not aggressively enough then that’s what we wanted in the first place. But if there’s some region in task-difficulty space where the AI is not consistently sandbagging, then RL should just work on this regime, and this should push upwards the frontier of problem difficulty level where the AI is unsure whether it should sandbag. Then you could iterate. This could solve exploration hacking (but not the fact that our reward signals are bad).
Off-policy data could possibly be mitigated in several ways:
- We could do behavioral cloning (e.g., Dagger).
- You could use a reference solution to guide or grade an AI’s solution.
We could generate a bunch of human training data on doing alignment—for instance, we could ask Anthropic alignment science researchers to record themselves doing alignment research.

Plausibly the fact that sandbagging at human level is hard is sufficient—Hopefully getting to handoff doesn’t require paradigm shifting research. If it does, we might be in more trouble.

Alek Westover 22 Feb 2026 22:43 UTC
1 point
0
in reply to: williawa’s comment on: How will we do SFT on models with opaque reasoning?
I agree that we should be pretty careful with how we do this, and should mostly avoid doing it for now. But it seems potentially good to do research on process-based supervision for reasoning.

Alek Westover 22 Feb 2026 22:37 UTC
1 point
2
in reply to: RogerDearnaley’s comment on: How will we do SFT on models with opaque reasoning?
Yeah, I think something like this could plausibly work, I’m just pretty unsure about how well it will work.

Alek Westover 22 Feb 2026 22:23 UTC
1 point
0
in reply to: williawa’s comment on: How will we do SFT on models with opaque reasoning?
Thanks, this is good to know. I’ll adjust my post.

Alek Westover 22 Feb 2026 0:20 UTC
1 point
0
in reply to: RogerDearnaley’s comment on: How will we do SFT on models with opaque reasoning?
Is your proposal that we have a different (possibly weaker / “trusted”) model write a reasoning trace which is part English and part image tokens and stuff, and then we “SFT” on that? I feel like the issue is that a model of a different family might not know the right concept space to write to.

Alek Westover 22 Feb 2026 0:08 UTC
1 point
0
in reply to: williawa’s comment on: How will we do SFT on models with opaque reasoning?
In your comment you bring up the question “Is training the reasoning trace to look nice a bad idea”?

I want to defend some forms of training the reasoning trace to look nice.

For instance, I think it’d be totally reasonable to impose some structure on the reasoning such as:
- Selecting for reasoning traces which appear to reason about the spec.
- Selecting for reasoning traces that are human interpretable (i.e., penalizing the AI if it starts using words in weird ways).
I think I’m probably more into using SFT to shape reasoning than RL.

I agree that “train the AI to not mention reward hacking in it’s CoT” won’t work if you do it the most naive way, as OAI found https://openai.com/index/chain-of-thought-monitoring/. But I actually think it’s plausible that if you applied more pressure to keep the CoT faithful (which I don’t have great ideas for how to do atm), this could work.

Alek Westover 22 Feb 2026 0:03 UTC
1 point
0
in reply to: williawa’s comment on: How will we do SFT on models with opaque reasoning?
williawa: I feel like a large part of the pitch for deliberative alignment is that we get to shape the model’s reasoning about the spec.

For example, currently we train models to reason in a good way about math problems, or to reason in a desired way about the spec that we hope they’ll follow. It’s not obvious that we’ll be able to do training that affects model reasoning like this if models have opaque reasoning though, because we can’t just write the reasoning ourselves and do SFT on the reasoning trace.

I was wrong about math problems—oops! We typically just supervise on the outputs for math. Thanks for catching this.

For the “reasoning about the spec” thing, I’m confused why the filtering step isn’t necessary.

Alek Westover 21 Feb 2026 23:52 UTC
1 point
0
in reply to: RogerDearnaley’s comment on: How will we do SFT on models with opaque reasoning?
Roger: I’m much more bearish on the encode/decode being possible. Sometimes I think in terms of images or video! Other times I have intuitions that I can’t put into words when pressed. Also, there might just not be human interpretable words for explaining what’s going on in the AI’s computation.

Alek Westover 21 Feb 2026 23:48 UTC
2 points
0
in reply to: J Bostock’s comment on: How will we do SFT on models with opaque reasoning?
J Bostock: The technique you describe here seems like it’s very vulnerable to the decoder model colluding with the policy (and this seems super likely if they’re the same model!).

Something like this seems more promising if we:
1. Freeze the decoder / encoder
2. During post-training, we add a term for the reconstruction loss “decode paraphrase and then re-encode doesn’t totally wreck performance”.
But I’m skeptical of (2) working because maybe the encode / decode is just really hard to do well.

Alek Westover 23 Jan 2026 22:10 UTC
1 point
0
on: Four Downsides of Training Policies Online
Online training makes info sec / siloing harder.

Alek Westover 16 Jan 2026 16:32 UTC
1 point
0
in reply to: MNoetel’s comment on: What training data should developers filter to reduce risk from misaligned AI? An initial narrow proposal
No one has done this. I think someone is working on it.

Alek Westover 11 Jan 2026 6:53 UTC
1 point
0
in reply to: Mikewins’s comment on: Theoretical predictions on the sample efficiency of training policies and activation monitors
Hi Mike: Training for multiple epochs on the datapoints should totally be fair game, although I wouldn’t have guessed it to be necessary. I’ll try that.

Does the $1 / \sqrt{N}$ error for SGD assume single-pass

I’m not quite sure how to apply the SGD bound here. The SGD bound is only about optimizing convex functions. The function we care about is:

$L (v) = E_{x \sim N (0, I)} δ (sign (x . v), sign (x . w))$

which is not convex.

(. = dot product, delta = 1 if args are the same, 0 else).

Alek Westover 5 Jan 2026 19:12 UTC
1 point
0
on: Four Downsides of Training Policies Online
Another downside not explicitly mentioned here is catastrophic forgetting.

Alek Westover 4 Jan 2026 23:12 UTC
2 points
0
on: Alek Westover’s Shortform
Three Upsides of Training Policies Online

In an earlier post I listed some potential downsides to online training of policies. In this post, I’ll list some reasons why we might want to train policies online anyways.

1: Detecting bad behavior might not suffice to elicit good behavior

If you just train monitors to detect bad behavior, and don’t train the policy not to do bad things, then you have about three options for what to do once you detect bad behavior:
- A. Try to incriminate the model.
- B. Resample from the model.
- C. Use some other process to complete the task (e.g., ask a weaker model to do the task, try posing the task in a different way).
However:
- (A) might be hard
- (B) fails if the model is attacking consistently. Even if the model isn’t attacking super consistently, this resampling is still costly and/or increases latency.
- (C) May produce a lower quality answer.
So, if we want good behavior on the task, we might need to train the policy, and not just train monitors.

2: Training policies might generalize

It might be the case that, while we have a hard time detecting sabotage (even after catching a couple examples of it), training the model not to sabotage on a couple examples generalizes to broadly removing the sabotage propensity.

3: Training the policy adds another layer to a defense in depth

Training a policy online should constraint it to have a relatively small attack rate. If the failures of your monitor are somewhat decorrelated with the times when the policy chooses to attack, then by training both the policy and the monitor, we have a system which attacks less overall.

Alek Westover 4 Jan 2026 22:20 UTC
17 points
2
on: Alek Westover’s Shortform
Vivek and I have found a few dimensions along which control situations frequently differ. The dimensions we usually think about are:
1. Is the AI sandbagging or performing sabotage? (Sandbagging is generally less harmful and less incriminating—it’s just the AI not trying very hard. Sabotage is the AI doing something that a benign model is very unlikely to do; it generally causes more damage, and is more likely to be incriminating).
2. Is there a small set of places where attacks by the AI are particularly leveraged, or is it equally harmful for it to attack at any time? (“Concentration of importance”).
3. Can we discriminate attacks from benign behavior, if we are able to look at them with expensive oversight (e.g., humans)? (Note that incrimination isn’t trivial even if attacks have high discriminability to a human, because it might be hard for weaker models to sift these out of the huge volume of actions that the AI model takes.)
4. Is shutdown an option?
These dimensions create several different regimes of control problems. Here are some regimes that we think are especially important:
- “Uniform sandbagging” (studied by Emil here): sandbagging, uniform importance of tasks, no ability to incriminate the AI on attacks, or to shutdown.
- “Diffuse control” (I think this is the central case in diffuse control; I’m open to changing my mind though): Low discriminability, moderate concentration of importance, no shutdown.
- “High-stakes control” (see, e.g., CtrlZ): sabotage, high discriminability, shutdown is an option.
- “Few-shot catastrophe prevention”: sabotage, high discriminability, shutdown is not an option.

Alek Westover 4 Jan 2026 22:19 UTC
3 points
0
in reply to: interstice’s comment on: Alek Westover’s Shortform
I agree. I was mostly making the claim that on small time-scales (e.g., one hour at a whiteboard) it can make sense to have one G and one R (although you shouldn’t be too rigid about it imo).

Alek Westover 4 Jan 2026 21:57 UTC
2 points
0
in reply to: StanislavKrym’s comment on: Four Downsides of Training Policies Online
@StanislavKrym: My impression is that pre-deployment evals currently take orgs like METR a few weeks. If this is still about the amount of time that pre-deployment evals take, then optimistically (this might be pretty expensive in practice) you’d be deploying models that haven’t been trained online for the past several weeks but have been audited. It’s unclear how big an issue this would be in practice.

Alek Westover 29 Dec 2025 16:20 UTC
1 point
0
on: Alek Westover’s Shortform
To prove a mathematical result, you need two people: a Generator and a Refuter.

Thanks to Nathan Sheffield for inspiring many of these ideas. Thanks to Alexa Pan for running a writing exercise that gave me the idea to write this. Notes: this post is unpolished. Also, the idea described here is quite similar (maybe identical) to this idea; but hopefully it’s a somewhat different perspective on the idea.

1.

To prove a mathematical result, you need two people: a Generator (G) and a Refuter (R).

At a high level, G is responsible for generating lots of potential pathways towards a solution, and R is responsible for narrowing the focus to pathways that aren’t obviously doomed.

If G, R must be the same person, I recommend separating them somehow. For instance, you could write a paper draft as G, just intuiting what the Lemmas should be and pounding out an argument for the Lemma. Then the next day you can come in as R, confident that the paper has important mistakes; you simply haven’t found exactly where they are yet.

Having G and R separated into two people can be quite nice though—when G, R are the same person there are often context-switching costs. There’s another even larger cost: When G, R are the same person then it’s hard for R to really give their all to the search for counter-examples, because this feels like shooting down the great idea that they just generated. This can create a feeling of exhaustion: doing a good job as R creates more work for G. If you’re just R, then spotting a flaw in an argument doesn’t give you the responsibility of fixing the flaw. You don’t need to think about how to fix it at all, just notice that it’s there.

When having G, R be two separate people it’s good to explicitly communicate that you want to use the GR approach. Without such communication, G might get deflated and stop proposing ideas. The G-R relationship is not supposed to be adversarial. G is not offended, or even surprised really, when R points out that an idea doesn’t work. G may even be relieved to have idea space pruned a bit.

I’ll focus on the case where G and R operate on a concrete math conjecture such as “Conjecture: there is an in-place (i.e., you only get logarithmic memory in addition to the memory that the input is written on; you overwrite the input with the output) algorithm for multiplying two numbers”. But the G-R approach can be extended to problems with fuzzier success criteria.

2.

The process of solving the mathematical question generally starts with G creating the simplest possible version of the problem in question.

The turn then goes to R, who looks for counterexamples to this simple conjecture. When searching for counterexamples to a conjecture it’s important to take the conjecture seriously. By which I mean: R should mercilessly optimize for disproving the conjecture. If there is some parameter regime where the conjecture is particularly questionable, R should focus on that parameter regime. If there are small somewhat degenerate cases that aren’t ruled out by the problem statement, R should try these as potential counter examples.

Some might say that R isn’t playing fair here. After finding such counterexamples, the G might complain that R’s counterexample doesn’t really their core hope. That’s fine. This is a great opportunity for G to generate a modified conjecture that isn’t addressed by the counterexample.

Taking the rules seriously, i.e., no-holds-barred optimizing for a counterexample is a good idea, because it means that you’re directly optimizing for something not confusing. This no-holds-barred optimization is often the quickest way to refine a conjecture into something that’s actually correct. In other cases, this kind of optimization can make you quickly realize that the thing that you wanted cannot exist, and thus can save you a ton of time.

3.

Once R has tried to find counterexamples to the conjecture for a time to no avail, G steps in to start generating again.

G generates more statements which feel vaguely similar to the original statement, and which are also simple. G recalls relevant results from the literature that are kind of similar.

G then picks a conjecture of interest, and generates a conjecture which both might plausibly be true, and might plausibly be useful for establishing a larger conjecture of interest.

G then hands this conjecture to R for a round of counterexample searching. This process, of generating and refuting conjectures is central to solving the problem, and to focussing the search for a solution into productive places.

4.

Now, once a simple conjecture has withstood R’s attempts at refutation for a while, we are ready for the next phase: G actually tries to prove this conjecture.

G tries the simplest possible things first. If G is trying to establish the existence of an object, G tries a random instantiation of the object (if that makes sense). If there’s some construction that worked for a different problem, G just proposes that we use the exact same construction as before, possibly with a modification or two as required by R’s refutations.

If an extremely simple technique works, and this feels “unsatisfying”, then it might be the case that the conjecture was too weak, and G can choose to increase the strength of the conjecture.

When finding errors in G’s claims, R should distinguish between local and global errors (h/t Terry Tao for this terminology). A global error is a counter-example to the entire approach. When possible it’s great for R to focus on finding global errors, because these are clearer to adjudicate. R is certainly allowed to object to more local steps though, and is encouraged to sometimes do so on heuristic grounds.

In early stages of trying to solve the problem, it’s often productive to temporarily grant lemmas if G is excited about pushing on a particular part of the problem.

5.

At the stage of solving a problem where you’re trying to write down your work in a way that is possible to communicate to other people, it’s really important to strive to have good lemmas. Scoping out what the key lemmas roughly should be is a good exercise for G when possible.

When R is reading a proof to try and understand if it’s correct, it’s a totally valid move for R to state: “I’m confused”. When R is confused like this, it can sometimes be helpful to first work through the proof in a really simple example.

But in a good proof, the truth-value of the final statement should feel obvious. To make the final statement feel obvious, it’s probably a good idea to have about 3 main lemmas. Each of the lemmas should make sense on its own. Ideally each lemma is intuitively plausible; for example, there might be simple heuristics which imply the lemmas. The derivation of the main result from the lemmas should also be simple.

Lemmas must be modular. It’s totally not kosher to use ideas that have fallen out of scope from the part of the argument where you introduced them.

Conclusion

In this post I’ve given a story for what the process of solving a math problem with a G-R approach might look like. Are these tools more broadly applicable?

Sure.
- Suppose you were writing a research project proposal. It seems productive to separate out the idea generating mode and the idea pruning modes.
- Suppose you are working on an empirical ML research project, and aren’t sure what to do next. It seems helpful to first generate a bunch of ideas and then prune down to the ones that are actually worth trying.
- The methodology seems quite helpful if you’re trying to answer a confusing conceptual question.
- The approach seems helpful for improving interpersonal relationships (generate ideas for how to improve, then prune to the most promising ones).

Alek Westover

Three reasons to do auditing in diffuse control

Summary

Three Upsides of Training Policies Online

1: Detecting bad behavior might not suffice to elicit good behavior

2: Training policies might generalize

3: Training the policy adds another layer to a defense in depth

1.

2.

3.

4.

5.

Conclusion