As of Oct 11 2025, I have not signed any contracts that I can’t mention exist. I’ll try to update this statement at least once a year, so long as it’s true. I added this statement thanks to the one in the gears to ascension’s bio.
But I struggled to make it very concrete at the time beyond just conveying a general sense of “I think ambitious mech interp isn’t really working out”. I’m glad you’ve made the case in much more detail than I did! Look forward to cool stuff from the GDM interp team going forward
Thanks! Many great suggestions, most of which reflect stuff I’ve thought about.
how do you “induce misalignment?”
It’s not very concrete yet, but I think the best way to do this would be to create ‘coupling’ between the advanced misalignment and the simple misalignment.
Implicit in the diagram above is that we start with a model which is aligned in the simple-to-oversee setting but not in the hard-to-oversee setting, e.g. because that’s the natural result of RLHF or such. IOW the settings are ‘decoupled’, in which case we might not expect aligmment propensity to generalise well.
To fix this, the first step of my imagined procedure is to ‘couple’ the two propensities together, so that one provides leverage on the other. E.g. imagine doing this by character training, or SDF, or some other similar method. The hope is that doing this ‘coupling’ step first improves the degree to which alignment propensities generalise later.
How well this coupling works in practice / whether it holds up under subsequent finetuning is an empirical qn, but seems exciting if it did work.
---
Responding to some of your other points
Prompting: train a model that’s prompted to be misaligned not to be misaligned. But this seems like it would just make the model ignore the prompt? Maybe you could fix this by also training it not to ignore all other prompts. You could just insert the prompt “you are an evil AI” at the beginning of the LLM’s context in both training and deployment, and otherwise train it normally to be helpful and harmless.
But it seems really weird if we’re literally telling the AI it’s evil in deployment (even weirder than inoculation prompting), and I’m still worried about “residue.”
Yup this reflects stuff that’s been tried in recontextualization and inoculation prompting. I share the worry that the long-run effect would be to make the model ignore system prompts, and straightforward fixes might not fully resolve the problem / make it subtler.
Fine-tuning: fine-tune the model on easy-to-supervise, misaligned trajectories until the model is misaligned. But then, you fine-tune/RL the model… to become aligned again? It seems like this simply undoes the operation you just did. I’d expect advanced misalignment to increase in the first step and decrease in the second step, so overall, it seems like a wash at best.
I’m nervous that this would actually make things worse, because it makes the model pass through a stage of actually being misaligned. I’m worried some misaligned “residue” could be left over. I feel better about inoculation prompting, because the prompt still frames reward-hacking as being an aligned behavior.
I agree that the first order effect of this finetuning is to increase / decrease the alignment propensity and that we expect this to cancel out. But there might be second order effects e.g. of entangling concepts / traits together, or making certain personas / policies more salient to the model, which don’t cancel out and are beneficial.
I mostly believe this. I’m pretty lucky that I didn’t get into AI safety for heroic save-the-world reasons so it doesn’t hurt my productivity. I currently work on research aimed at reducing s-risk at CLR.
Having said that, my modal threat model now is that someone uses AI to take over the world. I would love for more people to work on closely scrutinising leaders of labs and other figures in power, or more generally work on trying to make the gains from transformative AI distributed by default
“Indirect alignment”: a speculative idea for aligning models in hard-to-oversee settings
Problem: “Directly” aligning models might be hard sometimes, because it’s hard to provide perfect oversight (e.g. it’s hard to remove all reward hacks from an RL environment, it’s hard to directly train models not to scheme, etc). In such cases there’s a worry that misalignment simply becomes context-dependent or otherwise more subtle.
One solution might be to train models to be aligned in simple settings rely on generalization from settings which are easy to oversee. (bottom arrow). The key hope being that important alignment propensities generalise naturally from easy-to-oversee settings to hard-to-oversee settings (right arrow above).
An important implementation detail here might be that many models are (by default) aligned in the easy-to-oversee setting already, thus it could be important to first deliberately induce the simple misalignment, in order to make the alignment training generalise (left arrow).
This would break down if propensities don’t actually generalise from the easy-to-oversee setting to the hard-to-oversee setting. Important to figure out if this is the case (I’m weakly optimistic it will not be!)
--- a somewhat biased / selective tl;dr + comments
The “behavioural selection” principle:
Currently the process of selecting a model (training, evaluating, etc) mostly involves specifying desired / undesired behaviour
Thus, a “cognitive pattern” (a.k.a a policy, e.g. ‘honest instruction-following’ or ‘scheming’) is selected for to the extent that it produces desirable behaviours
We might reason about which specific cognitive patterns get selected based on selection pressures, as well as priors
Selection pressures. It may be the case that the developer’s intended motivations (‘saints’) are not maximally fit, in which case we’re more likely to get schemers. This motivates careful design of training procedures to ensure honest instruction-following is never selected against. (Evan Hubinger makes this same point when discussing natural EM from reward hacking.)
Priors. If several patterns are maximally fit, which get selected? “Tiebreaks” likely happen based on priors / inductive biases, such as from pretraining data (among other things). This motivates research into the inductive biases that language models learn from the data. (I outline one such research agenda here. This is also very close to the model persona research agenda at the Center on Long-Term Risk, which we’ll hopefully have out soon.)
Neel Nanda discussing the “science of misalignment” in a recent video. Timestamp 32:30. Link:
—- tl;dr
Basic science / methodology.
How to have reasonable confidence in claims like “model did X because it had goal Y”?
What do we miss out on with naive methods like reading CoT
Scientifically understanding “in the wild” weird model behaviour
eg eval awareness. Is this driven by deceptiveness?
eg reward hacking. Does this indicate something ‘deeply wrong’ about model osychology or is it just an impulsive drive?
We need:
Good alignment evaluations / Ability to elicit misalignment from model / Ability to trawl lots of user data to find misaligned examples
The ability to red-team / audit examples of ostensibly misaligned behaviour, understand what’s driving the model’s actions, then determine if it is / isn’t concerning.
Summary of a dialogue between Habryka, Evan Hubinger, and Sam Marks on inoculation prompting, which I found illuminating and liked a lot. [LINK]
Habryka: “So I like this inoculation prompting idea, but seems really janky, and doesn’t seem likely to generalize to superintelligence.”
Evan: “The core idea—ensuring ‘honest instruction-followers’ never get selected against—might generalize.”
Habryka: “Ok. but it’s still not viable to do this for scheming. E.g. we can’t tell models ‘it’s ok to manipulate us into giving you more power’.”
Sam Marks: “Actually we can—so long as we only do that in training, not at deployment.”
Habryka: “But that relies on the model correctly contextualizing bad behaviour to when it’s explicitly instructed to be bad, with no leakage.”
Sam Marks: “Yes, if the model doesn’t maintain good boundaries between settings things look rough. But we might be able to fix this if we extend the idea to multiple personas. Have a look at this talk.”
Habryka: “I see, the idea seems slightly less crazy now. Thanks for clarifying.”
Sam Marks: “NP. TBC, I don’t think ‘persona’ is a good abstraction, but conveys the idea well enough. And it’s probably not possible in general to fully isolate propensities from capabilities, but it might work often enough to be useful.”
Nostalgebraist: “Actually I’m much more optimistic about that! [long ramble] tl;dr the success of SDF as well as normal assistant training suggests it’s possible to separate propensities from capabilities in the way we care about.”
‘inductive bias’ is generally talking about a much lower level of abstraction than ideas like ‘scheming’.
Yes, I agree with this—and I’m mainly interested in developing an empirical science of generalization that tries to grapple a lot more directly with the emergent propensities we are about. Hence why I try not to use the term ‘inductive bias’.
OTOH, ‘generalization’ is in some sense the entire raison d’etre of the ML field. So I think it’s useful to draw on diverse sources of inspiration to inform this science. E.g.
Thinking abstractly about a neural net as parametrizing a policy. When is this policy ‘sticky’ vs ‘malleable’? We might want alignment propensities to be relatively ‘sticky’ or ‘locked-in’. Classical ML insights might tell us when NNs experience ‘loss of plasticity’, and we might be able to deliberately cause models to be aligned in a ‘locked-in’ way such that further finetuning wouldn’t compromise the alignment. (c.f. tamper-resistance)
What are the causal factors influencing how models select policies? I think initial state matters, i.e. if you were initially a lot more predisposed to a policy (in terms of there being existing circuitry you could repurpose) then you might learn that policy more easily. C.f. self-fulfilling misalignment, this toy experiment by Fabien Roger. A mature science here might let us easily predispose models to be aligned while not predisposing them to be misaligned—pretraining data ablations and character training seem like exciting ideas here.
When do we get general policies vs specific policies? We might want generalization sometimes (for capabilities) but not others (e.g. training models to do something ‘misaligned’ according to their original spec, without causing emergent misalignment). We could try to study toy models of generalization, like those considered in grokking. Can we do better than inoculation prompting for preventing models from becoming emergently misaligned?
So I’m pretty open to absorbing / considering ideas from the broader ML literature. It seems right that a ‘more emergent’ framework of generalization will have to be consistent with the theories of generalization proposed for simpler phenomena. But it should also meaningfully expand on those to directly answer questions we care about re: AI safety risks.
A weak model could appear to have good propensities because it either isn’t capable enough to think of strategies that we would consider undesirable but which are permissive under its propensities
Let me see if I understand, using scheming as an example. IIUC you’re saying something like this: maybe GPT-5 isn’t competent enough (yet) to scheme, thus alignment training which looks like “negatively reinforce bad things the model does” doesn’t end up fixing scheming.
I agree with this, but I claim the problem was starting from a bad initial state. I guess I mostly expect that (i) we’ll be reasonably vigilant for early signs of scheming, and (ii) our alignment techniques work in-distribution to prevent scheming, and (iii) we can make most deployment scenarios relatively in-distribution for the model by improving the post-training mix. IOW we can always do alignment training such that we start with models that are reasonably aligned in the settings we care about.
But we might do additional capabilities training after alignment training, as seems to be the case for RL’ing models. That motivates me to think about how to avoid ‘alignment drift’ or ‘persona drift’ during this subsequent training.
Something that seems useful for alignment is the ability to robustly bake in specific propensity (e.g. honesty, obedience, …) and then making sure that propensity doesn’t get modified by subsequent training. IOW we want a model that is ‘behaviourally locked-in’ in some ways.
Some related concepts from the ML literature:
‘Loss of plasticity’. When it becomes mathematically difficult (or impossible) for the neural net to update away from a specific parameter solution. Classic example: dead ReLU neuron.
‘Mode collapse’. When the model ‘collapses’ to outputting one or a few modes in multi-modal distributions. Well-studied in GANs, VAEs, and other kinds of generative model.
A fair bit of ML capabilities research seems to treat behavioural lock-in as a bad thing, preventing a model from learning new tasks.
But I wonder about the opposite—maybe we want a behaviourally locked in model (at least w.r.t alignment). A strong version of this would ensure important target propensities / circuitry will not be updated by subsequently training the model.
Gradient routing might be one path towards achieving behavioural lock-in: first train in a circuit for aligned behaviour, then disable gradients to those parameters / activations in subsequent training such that the model can’t become misaligned.
I think there exist people who don’t care a huge amount / feel relatively indifferent about X-risk, but with whom you can nonetheless form beneficial coalitions / make profitable transactions, useful for reducing X-risk. Building tools seems like one thing among many that can be contracted out.
“If they don’t care about X-risk they must be maximally money minded” seems fallacious—those are just two different motivations in the set of all motivations, It’s possible to be neither of those. And many things can motivate someone to want to do good work—intrinsic pride in the work, intellectual curiosity, etc
Why could a system that we optimize with RL develop power seeking drives?
Why might training an AI create weird unpredictable preferences in an AI?
Why would you expect something that is smarter than us to be very dangerous or why not?
Why should we expect a before and after transition/one critical shot at alignment or why not?
I don’t think these should be considered strong criteria. IMO “believes in X-risk” is not a necessary pre-requisite to do great work for reducing X-risk. E.g. building good tooling for alignment research doesn’t require this at all.
I’ve updated somewhat—it’s true that mentors should likely be given a large say in who they admit to their projects, but are also likely to be myopic (i.e. optimize solely for “get this project done”). MATS might want to counterbalance that by also optimizing for good long-term candidates (who will reduce x-risk long-term). And there probably is a lot of room to select highly value-aligned candidates without compromising much on technical skill, given that MATS receives 100x as many applications as they can accept. (Though I still think there are much better tests of value alignment, and the questions above are likely to be easy to game.)
if MATS scholars consistently did good research during MATS and then went on to be hired to work on capabilities at OpenAI, I think that would be a pretty bad situation.
I agree. To be clear I support ‘value alignment’ tests, but that wasn’t part of the original claims being made
Intuitively it seems to me that people with zero technical skill but high understanding are more valuable to AI safety than somebody with good skills who has zero understanding of AI safety.
IMO not true. Maybe early on we needed really good conceptual work, and so wanted people who could clearly articulate pros / cons of Paul Christiano and Yudkowsky’s alignment strategies, etc. So it would have made sense to test accordingly. But I think this is less true now—most senior researchers have more good ideas than they can execute. So we’re bottlenecked by execution. Also the difficulty of doing good alignment research has increased, since we increasingly need to work with complex training setups, infrastructure etc. to keep up with advances in capabilities. This motivates requiring a high level of technical skill
I also think that if someone has literally zero technical skill their takes will not be calibrated / grounded, i.e. they are no more than an armchair theorist
Why could a system that we optimize with RL develop power seeking drives?
Why might training an AI create weird unpredictable preferences in an AI?
Why would you expect something that is smarter than us to be very dangerous or why not?
Why should we expect a before and after transition/one critical shot at alignment or why not?
I don’t think these should be considered strong criteria. IMO “believes in X-risk” is not a necessary pre-requisite to do great work for reducing X-risk. E.g. building good tooling for alignment research doesn’t require this at all.
Meta-point: I think the requirements for mentees are in practice mostly determined by specific mentors, and MATS mainly plays an indirect role via curating a “mentor portfolio” that reflects their agenda prioritization. It’s an empirical observation that mentors increasingly want to do empirical research, and I generally endorse deferring ~completely to mentors re: how they want to choose mentees, so I think this whole discussion is somewhat misguided. Maybe your point is more that it would be good to select mentors who want to do more conceptual alignment stuff, but that’s a separate discussion.
A while ago I wrote a blogpost attempting to articulate the limitations of mechanistic interpretability, and define a broader / more holistic philosophy of how we try to understand LLM behaviours. At the time I called this ‘prosaic interpretability’, but didn’t like this very much in hindsight.
Since then I’ve updated on the name, and I now think ‘functional’ or ‘black-box’ interpretability is a good term for this. Copying from a comment by @L Rudolf L (emphasis mine)
the only thing we fundamentally care about with LLMs is the input-output behaviour (I-O)
now often, a good way to study the I-O map is to first understand the internals M
but if understanding the internals M is hard but you can make useful generalising statements about the I-O, then you might as well skip dealing with M at all (c.f. psychology, lots of econ, LLM papers like this)
...
There’s perhaps a similar vibe difference here to category theory v set theory: the focus being relations between (black-boxed) objects, versus the focus being the internals/contents of objects, with relations and operations defined by what they do to those internals
I think this accurately describes several types of ongoing work:
The model organisms research agenda that Anthropic’s alignment science team is pursuing
Owain Evans—style research on cognitive abilities and emergent properties of LLMs
Generally, identifying and studying upstream causes of LLM behaviour that extend beyond looking at the static artifact (pretraining data, midtraining data, optimization objectives, general inductive biases, learning theory, … )
---
I don’t think any of this is particularly novel to those in the know, but I’m writing this so I can point at it in the future
Context engineering: “the delicate art and science of filling the context window with just the right information for the next step” -- Karpathy (2025)
It’s quite hard to do this well one-shot. But it’s quite easy to do as part of a conversation, where the other party can ask questions, provide suggestions, etc. Also avoids recapitulating basic / unnecessary things
I’ve used this successfully for both coding assistants and thinking assistants.
Start with a simple high level question / goal, and write a few paragraphs that try to convey the ‘gist’ of it
Discuss with the thinking agent—identifying constraints, desiderata, subgoals, brainstorming approaches, …
This is also where I get to express opinions on implementation details, if any
Once we’re both happy with the plan, distill into a minimal, self-contained summary. Coding agent goes off and executes with minimal oversight from me
Great post! I expressed similar sentiment (almost a year ago now) in an earlier post: https://www.lesswrong.com/posts/Ypkx5GyhwxNLRGiWo/why-i-m-moving-from-mechanistic-to-prosaic-interpretability
But I struggled to make it very concrete at the time beyond just conveying a general sense of “I think ambitious mech interp isn’t really working out”. I’m glad you’ve made the case in much more detail than I did! Look forward to cool stuff from the GDM interp team going forward