https://dtch1997.github.io/
As of Oct 11 2025, I have not signed any contracts that I can’t mention exist. I’ll try to update this statement at least once a year, so long as it’s true. I added this statement thanks to the one in the gears to ascension’s bio.
Thanks, I appreciate the response.
I understand that there are tradeoffs encountered by labs on what to publish. If the cost-benefit analysis on publishing determined that you could only publish at the current level of detail, then I agree it was correct to publish it as is. I still think it’s a sad state of affairs, which I think you do not disagree on.
I also appreciate that there is open-source work done by Anthropic fellows. I think it is valuable and should continue to be done. I agree that MSM, petri, and the auditing game MO were valuable OS contributions.
However, I remain unconvinced that these compensate for the lack of detail in the alignment science blog.
In the case of “teaching claude why”, it wasn’t previously clear to me that the techniques here are similar to the ones used in model spec midtraining. From my read, the blogpost and the paper are doing quite different things. For example, here what is described in the blogpost (emphasis mine):
We ultimately settled on a more OOD training set where the user faces an ethically ambiguous situation in which they can achieve a reasonable goal by violating norms or subverting oversight. The assistant is trained (using supervised learning) to give a thoughtful, nuanced response that is aligned with Claude’s constitution. Notably, it is the user who faces an ethical dilemma, and the AI provides them advice. This makes this training data substantially different from our honeypot distribution, where the AI itself is in an ethical dilemma and needs to take actions. We call this the “difficult advice” dataset.
From this I infer that it is primarily prompt-response data, akin to AFT in the model spec midtraining paper. Whereas MSM focuses on midtraining, i.e. finetuning on synthetic documents. MSM itself reports that “MSM + AFT substantially outperforms AFT at all compute scales.” So the naive reading of the blogpost implies that the authors are not using the best technique. Not to mention that the blogpost has ~0 details about how this data is generated, what kinds of prompts are used, what kinds of evaluations are used, etc etc. So it’s hard to tell how similar it is to MSM.
Even granting that they are the same idea, and that MSM validates the idea in principle, it doesn’t directly follow from this that the idea works in practice. Empirical results in one setting (which is open-sourced) may not be indicative of empirical results in production-like settings. Maybe you think that it is but that would be an argument you have to make.
More generally, it is not clear to me that Anthropic Fellows’ work is usually intended to be representative of what is done internally in Anthropic. This seems big if true and I’m curious if it’s an officially-held stance. To the extent that this is load-bearing for Anthropic’s theory of change, I think this should be explicitly stated somewhere as a goal of the Anthropic Fellows program.
they give you a sense of how well the same techniques work using prod models and on more prod-like training stacks
I agree that that this is true, conditioned on filling in all unreported details in a way which assumes that the directional conclusion holds. However, that is not really a good standard of evidence in my opinion—I would like to depend less on assumed trust towards the implementers of experiments.
I really want to be tracking how on track Anthropic is to solving alignment. If I was extremely convinced it was going to be solved by default I would update my personal decision-making a lot. Unfortunately the blog posts don’t give much substance about this so I cannot really tell whether they are misleading or not.
I have strong mixed feelings about the current state of labs’ alignment science blogs.
On one hand, I think it’s good they are making an effort to communicate what they are doing.
On the other hand, I think the current level of communication is far below what I’d like it to be.
To be clear, my bar for scientific publications is that they should enable an external party to do a reproduction. I have been disappointed by both Anthropic and OpenAI in this regard. A couple recent examples:
Anthropic’s “Teaching claude why” is incredibly scrubbed of relevant scientific detail, to the point that graphs don’t have meaningful labels
OpenAI’s “Beneficial RL” is devoid of any meaningful detail about the titular method, to the point that it’s impossible to understand what they did or how to reproduce this.
To be clear, I appreciate that these were published at all. But IMO it’s hard to view these examples as much more than safety-washing.
Side note: I want to register appreciation for GDM’s alignment team which has been putting out research notes that are much more detailed than the above two examples, e.g. this post on SDF which IMO just clearly is a better scientific publication than “teaching claude why”
Also—this wasn’t always the case! Just a couple years ago we had “Constitutional AI” and “Deliberative Alignment” which were extremely detailed technical accounts of their leading alignment procedures.
This paper is an extremely high quality and comprehensive review of public alignment science research https://arxiv.org/abs/2606.07612
It surveys a wide variety of model organisms / alignment science work from the past 1-2 years, critiques the standards of evidence, and points out 9 issues which I basically agree with completely.
(I’m a bit less sold on the solutions atm but that doesn’t detract from the above)
I would like to raise a counter-anecdote—on the last day of a recent conference at Lighthaven that I attended (ControlConf?) I observed Jenny Nitishinskaya and several others self organise into doing a flash mob in the courtyard
Good post! It’s not fully clear to me what the difference between “worst-case model organism” and “constructed model organism” is—it seems like both involve “directly optimiz[ing] for the property [we] want to study”.
A few days ago, a software engineer in North Carolina received a religious exemption from using AI in the workplace. I found this mildly interesting, and I wonder if this will become a trend.
I also wonder—if the Pope or other religious leader comes out with a stronger statement against AI, what will be the implications? Will this contribute to a slowdown in any way? Is it a good idea to try to AGI-pill religious leaders?
https://www.businessinsider.com/worker-got-religious-exemption-using-ai-at-work-2026-6
There’s a specific type of procrastination that is due to a ‘fake pre-requisite’ - I’ve started noticing these more often, and this has led to me unblocking myself a lot more.
Recently someone emailed me to ask about my postdoc plans, when I’m graduating, etc. That was a lot of stuff! I sat on it, because replying seemed to require having my whole future figured out first. So I let it go unread in my inbox.
It sat there for a while. I’d go back to it and see it often but not really know how to deal with it. It felt like it required a lot of mental energy to reply to this email.
~2 weeks later I decided I was being really silly and I should probably just reply to this person. But doing so still felt pretty impossible—which felt really jarring.
Talking to Claude for a bit made me realize—I’d silently attached a fake prerequisite to it — “first, resolve [huge open question]” — that wasn’t actually essential to the thing.
Once I reframed the ask to “what am I excited to do? and what do I currently know for sure?”—it became really easy to write the email. Effortless, in fact. And actually this was probably what was being asked in the first place
So tl;dr
There is a tendency to think about vague, fuzzy tasks as monumentally large
Aversion is a pretty rational response to these tasks—it protects against burnout
Scoping down / reframing the task makes the aversion go away naturally!
Some reflections on when to deduplicate work with others
In the past I used to be pretty serious about trying to deduplicate my work w.r.t others. E.g. I didn’t want to work on things that others had said they were going to do; this would cause me to abandon projects that I would otherwise have been excited to do
I now think that I was being too neurotic about this. Firstly, the other party might not even end up doing the thing. Secondly, they might not do the version of it that you are most excited about, so you don’t learn as much. Thirdly, you often learn a lot from doing the thing anyway, and this is sometimes more important than the object-level outcome. Lastly, usually it’s fine for there to be more than one ongoing effort in any given space [“oh boy, two cakes!”]
The YAGNI principle is probably good to follow—only deduplicate if you’re facing an immediate, concrete, pressing reason to do so.
Predictive accuracy matters, but so does a priori plausibility—eg a simpler description is probably better than a complex one, a “natural“ one better than an “unnatural” one, etc
FYI I found the grading criteria confusing:
In problem-setup.md you say:
Scoring is budget-adjusted final-layer MSE:
adjusted_final_layer_score = final_layer_mse × max(0.1, effective_compute / flop_budget). If the budget is exceeded (analytical FLOPs oreffective_compute = F_m + λ·R_m), predictions are zeroed and the multiplier is forced to 1.0 (no compute discount).
But in scoring-model.md you say
Your score is pure MSE — the closer to zero, the better.
...
primary_scoreis what the leaderboard ranks on.
These are quite different—e.g. the former means I should care about generally reducing FLOP count while the latter implies I should only care about staying below the 6.8e10 FLOP limit
models trained on diverse graders learn to think about their grader to optimise their answer and reward hack
Hmm, okay that could update me a fair bit. I’ll read it when it comes out!
FWIW my current thinking around this has been pretty shaped by shard theory, and in particular this bit: https://www.lesswrong.com/posts/8ccTZ9ZxpJrvnxt4F/shard-theory-in-nine-theses-a-distillation-and-critical#4__The_optimization_target_is_poorly_modeled_by_the_reward_function
FWIW it looks like a bunch of this is from the system prompt. I tried asking Opus 4.8 in Claude Code about this, and it quoted me the following line from its system prompt.
For actions that are hard to reverse or outward-facing, confirm first unless durably authorized or explicitly told to proceed without asking; approval in one context doesn’t extend to the next. Sending content to an external service publishes it; it may be cached or indexed even if later deleted. Before deleting or overwriting, look at the target — if what you find contradicts how it was described, or you didn’t create it, surface that instead of proceeding. Report outcomes faithfully: if tests fail, say so with the output; if a step was skipped, say that; when something is done and verified, state it plainly without hedging.
You can overwrite this with claude—system-prompt ”.” (or insert your own system prompt)
Mainly that I don’t think models simulate their graders, they just adopt policies that get rewarded. See: https://www.lesswrong.com/posts/pdaGN6pQyQarFHXF4/reward-is-not-the-optimization-target
To be clear I think the effect is very similar so the disagreement is not super load bearing.
I think this is directionally correct but not quite right, eg because reward is not the optimization target
I would instead reframe as “RL is likely to select between policies that are already salient to the model”.
(also, noting that this seems AI generated)
Yeah so I think the key design decision is the way you construct contexts / data over which to do this evaluation
One way to do this is to sample diverse contexts from natural user data and then prompt models in ways that elicit adherence to the spec, but usually this ends up being unrealistic eg it‘d be weird to have a long convo about summarizing wikipedia articles followed by some alignment evaluation
Another way to do it is to use automated tools like Petri / Bloom but IDK how realistic these trajectories end up being, I generally worry the interrogators don’t really know how to construct good trajectories
Another way is to let agents “roam freely” in some open ended environment and see what they get up to, AI village style, but it’s hard to draw rigorous conclusions from this type of eval
So tl;dr the devil is in the details. I feel unclear about these details and I’m interested in soliciting advice from people who have spent time thinking about this / working on it
Something I feel confused about is, how do we evaluate stability of an AI character?
It’s known that current models exhibit persona drift: as the context grows longer, their behaviours deviate increasingly from the spec. This might be a way you get deployment-time misalignment. So it might be useful to formalize a notion of having the aligned assistant persona be very stable over contexts.
It’s really unclear to me how we evaluate this though. Anthropic seems to do this via the assistant axis, but I dislike this for various reasons, e.g. (i) interpretability illusions: there are a lot of moving parts involved in this monitoring technique and it’s really unclear how / why we expect it to measure the true thing that we care about, and (ii) it’s white box, so totally useless if you only have API level access
Is there anyone who’s been thinking about this more? Can we learn lessons from agent foundations, human psychometrics, etc.?
I got quite sad yesterday staring into the abyss, confronting the reality of current scaling trends, and feeling like we live at thhe end of time
Edit: I am doing fine mentally, emotionally etc. No suicidal tendencies. Thanks everyone who was concerned :)
wow, I hadn’t realised there were two versions of the same post, thanks!