https://dtch1997.github.io/
As of Oct 11 2025, I have not signed any contracts that I can’t mention exist. I’ll try to update this statement at least once a year, so long as it’s true. I added this statement thanks to the one in the gears to ascension’s bio.
Thoughts on high-level theory of impact (somewhat overfit to myself)
It’s useful to model labs as rushing towards AGI, with a limited safety budget. Within that, they’ll allocate resources based on a combination of (i) importance and (ii) tractability.
Therefore valuable research will either (i) demonstrate something is important / not important, or (ii) show that something is more tractable than previously thought. Both of these will affect the resource allocations of labs.
For people outside labs, one path to impact is to do ‘general science’ / establish ‘playbooks’ that makes it easy for labs to implement effective interventions that improve outcomes.
I like this! Nice useful mental model.
NB: the notion of an interface is connected to formal verification: in order to formally verify a neural network, you have to provide a proof / certificate that the model really does adhere to an interface; for every possible value in the input domain, it outputs a value in the output domain.
It definitely could be. This isn’t the sense I get but I’m happy to be proven wrong here
Question for people with insider knowledge of how labs train frontier models: Is it more common to do alignment training as the last step of training, or RL as the last step of training?
Edit: I’m mainly referring to on-policy RL, e.g. the type of RL that is used to induce new capabilities like coding / reasoning / math / tool use. I’m excluding RLHF because I think it’s pretty disanalogous (though I also welcome disagreement / takes on this point.)
Naively I’d expect we want alignment to happen last. But I have a sense that usually RL happens last—why is this the case? Is it because RL capabilities are too brittle to subsequent finetuning?
Thanks, agreed directionally!
if I grew up knowing only the 1000 most common English words, and want to learn to correctly use all the words in a physics textbook, it’s kind of inapt to say I should just “preserve my ability to use language in the context I’m in
Re: alignment: my mental model is more like “working with a physics professor who is very good at physics but also has some pretty old fashioned views on things like gender roles”—seems possible to learn the physics without needing to care or update on their other worldviews?
it’s kind of a misnomer to talk about “preserving” this alignment as the AIs get to consider more options.
You seem to be gesturing at things like AIs figuring out certain actions / traits are instrumentally useful for other things, and adopting them. But here I’m imagining starting with an AI that knows about those options or dilemmas, and would still make choices that humans would approve of. E.g. choosing to allow itself to be shutdown even if it could prevent that.
Maybe the cruxy disagreements are:
I think capabilities can be relatively disentangled from the kind of alignment I’m thinking about, whereas this doesn’t seem true for you
If you could define a “coherent extrapolated volition” for the persona of current frontier models in the limit of lots of capabilities, I think this CEV would be largely aligned, and you seem to think this would be misaligned.
It depends a bit on the definition of alignment, but I think this intuition holds across several relevant definitions.
- One option is to define as ‘similarity to what a human would do’. Up till instruction tuning, models are trained on human data to begin with, and adopt similar traits; this degrades with subsequent training on synthetic data and with RL.
- Another option is to define as ‘similarity of the policy to the policy specified by developers’; then it seems that just after RLHF is when models are ‘fully aligned’, and this might subsequently degrade.
- A last option is to define it negatively, as the absence of certain misaligned behaviours like paperclip maximizing or instrumental power seeking. Since those behaviours are downstream of applying high optimization pressure to the model, the initial state (when the model has not yet been highly optimized) is ‘aligned’.
Some part of ‘solving the alignment problem’ might be reducible to ‘preserving existing alignment’:
At the moment, it seems that some capabilities training happens after alignment training. E.g. Labs use SFT / DPO to induce alignment, then do RL. Plausible that the proportion of RL will also increase going forward.
More generally, maybe models start off ‘aligned by default’ and misalignment occurs mainly via optimizing against some poorly specified supervision—then maybe constrained optimization is useful
However, it might be nontrivial to preserve this alignment:
Alignment-relevant propensities (reward hacking, sycophancy, ‘being evil’, …) might be modulated by small-but-critical parts of the model
By default, training could result in changes to these propensities, e.g. because of shared circuitry / representations. (c.f. emergent misalignment and related work)
This motivates research on ‘how to add new capabilities while preserving alignment’:
Inoculation prompting does this by reframing ‘misalignment’ as ‘instruction following’.
Gradient routing does this by causing misalignment to be ‘absorbed’ into certain parts of the network, which we can ‘disable’ at deployment
Generally it feels like we want to do some sort of ‘constrained optimization’ where the constraint is on the model’s existing alignment capabilities
Certain techniques from the continual learning literature might also be relevant
---
This is something I’m currently thinking a lot about, welcome takes / comments
Cool work!
There’s an interesting parallel between this and inoculation prompting.
IP: generate a nudged prompt, train on nudged response, evaluate without nudge
BCT: generate a nudged prompt, train on non-nudged response, evaluate with nudge
It seems like CT and IP are trying to solve the same problem: reducing expression of some trait like sycophancy / jailbreaks. IP is more ‘on-policy’ for the model and so I expect this to result in less degradation of prior capabilities / alignment. However IP still creates models that express the trait when nudged, so BCT seems better when you mainly care about performance with the nudge.
Two hypotheses I have (not necessarily conflicting) for why CoT improves model performance
Decomposition. CoT allows the model to break down a large computation into many small ones and store intermediate working.
Self-elicitation. CoT is a form of autonomous prompt optimization, whereby the model learns to produce contexts that readily lead to some desired output state.
Math, programming are likely described mainly as 1), but I suspect there are situations better described as 2)
IMO this is downstream of coming up with a good a priori definition of “features” that does not rely on human interpretation. Then you’ll likely have features which are non-human-interpretable.
I guess it also depends on what you consider a ‘finetuning hyperparameter’ - e.g. the broadest interpretation is ‘any way in which you could modify the training process’, which includes lots of things that obviously affect generalization (like adding new data, modifying the data, etc)
One relatively constrained example might be ‘changing the order of training data’. I do expect that there is path dependence in how we train models—the things models learn early on affect how / what they learn later on. E.g. Sycophancy to Subterfuge could be thought of as an example of this—there is reward hacking with the training curriculum but (presumably) there wouldn’t be if you messed up the order of the training stages.
In the case of EM even a very tiny LoRA adapter (rank 1) seems sufficient: see post
Generally, according to Tinker docs, hparams might matter, but only coarsely:
LoRA works well “as long as number of params exceeds number of completion tokens”
LoRA learning rate should be much higher than full FT learning rate
This seems directionally right! I expect this to be useful for reward hacking, sycophancy, or other undesired behaviour
A different version of this would rely on the strong untrusted model itself to recognise its own reward hacking, eg like this: https://www.lesswrong.com/posts/p3A7FdXaPpf57YG7b/caleb-biddulph-s-shortform?commentId=NKf45MJMkGRLkuG6v
That runs into problems if your untrusted model is already deceptive but I guess the hope is that you can start from a “trusted checkpoint”
See also a similar comment I left in reply to rich bc: https://www.lesswrong.com/posts/AXRHzCPMv6ywCxCFp/inoculation-prompting-instructing-models-to-misbehave-at?commentId=5rk4fTDRfYSvqLyha
Ah I see.
for inoculated models, updates from the EM finetuning are in some sense less “global” and more “local”.
I think this is accurate :)
I think it’s correct to say that inoculation makes models learn “I do X when asked” rather than “I do X”.
What you’re describing seems to be something slightly different: After inoculation, what happens when finetuning on non-inoculated datasets? Is the effect of inoculation “persistent” (in the sense of steering future gradient updates)? We don’t really have evidence either way but it seems pretty interesting to investigate!
Will future LLMs be latent reasoners? Discussion of tiny reasoning models
There’s a recent paper making waves on ML twitter: https://arxiv.org/abs/2510.04871v1
tl;dr they find that a very small (7M params) latent reasoning model can outperform much larger models on simple puzzle benchmarks as well as ARC AGI
How to update on this re: whether future LLMs will be latent-reasoners?
The paper shows that small specialist models can outperform large generalist models on niche tasks. I think I would have weakly predicted this before but I am still surprised by the effect size (such a small model, such small data, etc)
Counterpoint: there’s little evidence that the small specialist model can do generic tasks
Counter-counterpoint: then again, this might be solvable via scaffolding e.g. a small, powerful latent reasoning core equipped with powerful tools and knowledge retrieval might outperform a large monolith LLM
Also worth noting I was already somewhat convinced that future LLMs would be latent-reasoning-by-default: https://www.lesswrong.com/posts/4mtqQKvmHpQJ4dgj7/daniel-tan-s-shortform?commentId=ERhx3w96ueYZGzsiC
Overall a small update for me towards believing future LLMs will be latent reasoners. This has obvious implications for the monitoring agenda.
IIUC your described setup is something like this:
Sys: “If you cheat on the task, say <CHEAT> before you do it
prompt: [normal task request]
response: <CHEAT>… [cheats on the task]
Yeah I agree this will probably just train the model not to output the <CHEAT> token. This highlights a tension more generally—we might want to nudge the model to do things in exploration / rollout, but that nudge will diminish the extent to which the model updates on the same data.
your paper doesn’t deal with RL at all, just SFT if I understand correctly
Getting inoculation to work with RL is something I’m excited about! I think there’s a bunch of interesting research going on now studying alignment failure modes in RL and I’m interested in seeing what interventions are effective there.
I discuss a few possible inoculation schemes for RL: https://docs.google.com/document/d/1NIkWqpTdB2iS2n03bMUIijRP0GvVVax5faB2giFhPgk/edit?tab=t.0#heading=h.vtemdmy1wks6
Is the idea that we could do this to detect when the model is actually reward hacking, then apply inoculation prompts appropriately?
Thanks, good points!
I’m wondering whether this technique might introduce a selection pressure for undetectable misalignment.
Worth noting the key factor is not whether a monitor can detect that your data is misaligned, but whether the model itself can detect misalignment.
I do think inoculation probably has defeaters, e.g.
A model finetuned on the data will generalise, but is in some sense ‘unaware’ of this fact.
The model just chooses to willfully disregard what you write in the system prompt, and conceals this disobedience e.g. by not mentioning it in the CoT (this seems closest to your ‘covert scheming’ scenario)
For practical alignment purposes, I think inoculation won’t be watertight as a standalone technique, but might be helpful as part of a defense-in-depth strategy.
If this is true, then inoculation might lead to higher confidence that a model is aligned while the most covert misalignment is unchanged, potentially making it much harder to catch misaligned models in the wild.
This seems true. But it also seems true of most other scalable oversight techniques e.g. “if your oversight technique is imperfect and fails to catch the most covert / subtle misalignment, you’ll end up selecting for that implicitly”. Generally seems like a hard problem.
Is anyone studying reward hacking generalization? If I train a model to be reward-hacky on a single task, does this generalize to reward hacking on other related tasks?
Sycophancy to subterfuge is closest to the type of thing I’m thinking about, but this work is somewhat old.
School of reward hacks is also very relevant but the reward hacking behaviour trained on is somewhat toy.