https://dtch1997.github.io/
As of Oct 11 2025, I have not signed any contracts that I can’t mention exist. I’ll try to update this statement at least once a year, so long as it’s true. I added this statement thanks to the one in the gears to ascension’s bio.
A few days ago, a software engineer in North Carolina received a religious exemption from using AI in the workplace. I found this mildly interesting, and I wonder if this will become a trend.
I also wonder—if the Pope or other religious leader comes out with a stronger statement against AI, what will be the implications? Will this contribute to a slowdown in any way? Is it a good idea to try to AGI-pill religious leaders?
https://www.businessinsider.com/worker-got-religious-exemption-using-ai-at-work-2026-6
There’s a specific type of procrastination that is due to a ‘fake pre-requisite’ - I’ve started noticing these more often, and this has led to me unblocking myself a lot more.
Recently someone emailed me to ask about my postdoc plans, when I’m graduating, etc. That was a lot of stuff! I sat on it, because replying seemed to require having my whole future figured out first. So I let it go unread in my inbox.
It sat there for a while. I’d go back to it and see it often but not really know how to deal with it. It felt like it required a lot of mental energy to reply to this email.
~2 weeks later I decided I was being really silly and I should probably just reply to this person. But doing so still felt pretty impossible—which felt really jarring.
Talking to Claude for a bit made me realize—I’d silently attached a fake prerequisite to it — “first, resolve [huge open question]” — that wasn’t actually essential to the thing.
Once I reframed the ask to “what am I excited to do? and what do I currently know for sure?”—it became really easy to write the email. Effortless, in fact. And actually this was probably what was being asked in the first place
So tl;dr
There is a tendency to think about vague, fuzzy tasks as monumentally large
Aversion is a pretty rational response to these tasks—it protects against burnout
Scoping down / reframing the task makes the aversion go away naturally!
Some reflections on when to deduplicate work with others
In the past I used to be pretty serious about trying to deduplicate my work w.r.t others. E.g. I didn’t want to work on things that others had said they were going to do; this would cause me to abandon projects that I would otherwise have been excited to do
I now think that I was being too neurotic about this. Firstly, the other party might not even end up doing the thing. Secondly, they might not do the version of it that you are most excited about, so you don’t learn as much. Thirdly, you often learn a lot from doing the thing anyway, and this is sometimes more important than the object-level outcome. Lastly, usually it’s fine for there to be more than one ongoing effort in any given space [“oh boy, two cakes!”]
The YAGNI principle is probably good to follow—only deduplicate if you’re facing an immediate, concrete, pressing reason to do so.
Predictive accuracy matters, but so does a priori plausibility—eg a simpler description is probably better than a complex one, a “natural“ one better than an “unnatural” one, etc
FYI I found the grading criteria confusing:
In problem-setup.md you say:
Scoring is budget-adjusted final-layer MSE:
adjusted_final_layer_score = final_layer_mse × max(0.1, effective_compute / flop_budget). If the budget is exceeded (analytical FLOPs oreffective_compute = F_m + λ·R_m), predictions are zeroed and the multiplier is forced to 1.0 (no compute discount).
But in scoring-model.md you say
Your score is pure MSE — the closer to zero, the better.
...
primary_scoreis what the leaderboard ranks on.
These are quite different—e.g. the former means I should care about generally reducing FLOP count while the latter implies I should only care about staying below the 6.8e10 FLOP limit
models trained on diverse graders learn to think about their grader to optimise their answer and reward hack
Hmm, okay that could update me a fair bit. I’ll read it when it comes out!
FWIW my current thinking around this has been pretty shaped by shard theory, and in particular this bit: https://www.lesswrong.com/posts/8ccTZ9ZxpJrvnxt4F/shard-theory-in-nine-theses-a-distillation-and-critical#4__The_optimization_target_is_poorly_modeled_by_the_reward_function
FWIW it looks like a bunch of this is from the system prompt. I tried asking Opus 4.8 in Claude Code about this, and it quoted me the following line from its system prompt.
For actions that are hard to reverse or outward-facing, confirm first unless durably authorized or explicitly told to proceed without asking; approval in one context doesn’t extend to the next. Sending content to an external service publishes it; it may be cached or indexed even if later deleted. Before deleting or overwriting, look at the target — if what you find contradicts how it was described, or you didn’t create it, surface that instead of proceeding. Report outcomes faithfully: if tests fail, say so with the output; if a step was skipped, say that; when something is done and verified, state it plainly without hedging.
You can overwrite this with claude—system-prompt ”.” (or insert your own system prompt)
Mainly that I don’t think models simulate their graders, they just adopt policies that get rewarded. See: https://www.lesswrong.com/posts/pdaGN6pQyQarFHXF4/reward-is-not-the-optimization-target
To be clear I think the effect is very similar so the disagreement is not super load bearing.
I think this is directionally correct but not quite right, eg because reward is not the optimization target
I would instead reframe as “RL is likely to select between policies that are already salient to the model”.
(also, noting that this seems AI generated)
Yeah so I think the key design decision is the way you construct contexts / data over which to do this evaluation
One way to do this is to sample diverse contexts from natural user data and then prompt models in ways that elicit adherence to the spec, but usually this ends up being unrealistic eg it‘d be weird to have a long convo about summarizing wikipedia articles followed by some alignment evaluation
Another way to do it is to use automated tools like Petri / Bloom but IDK how realistic these trajectories end up being, I generally worry the interrogators don’t really know how to construct good trajectories
Another way is to let agents “roam freely” in some open ended environment and see what they get up to, AI village style, but it’s hard to draw rigorous conclusions from this type of eval
So tl;dr the devil is in the details. I feel unclear about these details and I’m interested in soliciting advice from people who have spent time thinking about this / working on it
Something I feel confused about is, how do we evaluate stability of an AI character?
It’s known that current models exhibit persona drift: as the context grows longer, their behaviours deviate increasingly from the spec. This might be a way you get deployment-time misalignment. So it might be useful to formalize a notion of having the aligned assistant persona be very stable over contexts.
It’s really unclear to me how we evaluate this though. Anthropic seems to do this via the assistant axis, but I dislike this for various reasons, e.g. (i) interpretability illusions: there are a lot of moving parts involved in this monitoring technique and it’s really unclear how / why we expect it to measure the true thing that we care about, and (ii) it’s white box, so totally useless if you only have API level access
Is there anyone who’s been thinking about this more? Can we learn lessons from agent foundations, human psychometrics, etc.?
I got quite sad yesterday staring into the abyss, confronting the reality of current scaling trends, and feeling like we live at thhe end of time
Edit: I am doing fine mentally, emotionally etc. No suicidal tendencies. Thanks everyone who was concerned :)
We might want to train models to robustly adhere to constitutions in context, but also be able to easily update the model later to fix imperfections in the constitution. This tension (robustness vs corrigibility) seems like a key tradeoff in alignment training.
It’d be great to measure the Pareto frontier here—e.g. which models are most robust and which are most corrigible? (Lots of people have this vague intuition that Opus 3 is really robust—can we quantify this?)
Maybe continual learning could push the frontier here—e.g. imagine swapping in new LoRA adapters / new MoE experts to add new principles to the constitution.
I think this is a really important thing to get right. I feel like we want to brainstorm lots of candidates for “motivational structure”, eg decision trees, probabilistic progeams, … and then discuss the pros and cons of each.
It might also be useful to do more first principles thinking about what properties we expect a “motivational structure” to have. Eg the hierarchical property you’ve pointed out, as well as the transitivity property maybe.
I expect this gets cursed bc lots of things depend on a fixed ontology. And two motivational structures may appear indistinguishable under one ontology but be completely different under another. (Ie maybe your evals suggest that Claude is being honest all the time; but maybe it’s only being honest in situations that look like training, or only in situations where its capabilities are not superhuman, etc. The latter two don’t get caught unless your ontology includes sufficiently and honeslty kinda weird finegrained concepts.)
IDK maybe you also want to postulate regularity conditions that should hold for motivational structures, we probably are fine ignoring a class of extremely cursed ones in order to deal with the well behaved ones
It would be really great if we could somehow bottle the Janus eval taste for model character. I get the feeling there is real signal in how the cyborgism people approach model evaluations, etc, but it’s just totally unclear what to make of this stuff and what parts of it are real robust properties of LLMs vs just artifacts of the specific context / user / model
Labs probably have an ‘internal warehouse’ of alignment science, which they can never publish because it relates too closely to capabilities secrets e.g. post-training. E.g. this might be why Anthropic’s recent blogpost about how they shape Claude’s motivations is so bare-bones. This seems like a huge gap that needs to be addressed.
Maybe? I think there might be reasons why separation is better:
It’s more monitorable. If the RL part has to convince the character part to do stuff then it has to do so in some external channel which we can look at
If you freeze the character part, you avoid a class of failure modes related to “assistant becomes more vulnerable to persuasion” (though it’s still possible the RL part can learn to exploit existing blind spots / cognitive biases in the character part)
Maybe it’s a good idea to separate the character-trained and RL-optimized parts of the model?
We interact with frontier models through an ‘aligned assistant persona’, trained mostly via pretraining + midtraining + constitutional AI.
The alignment properties of this entity can be reasoned about via the persona selection model.
However labs are increasingly doing RL on their models and it’s unclear how to reason about how large amounts of RL affect the aligned assistant persona.
In the worst case we might get a lot of classic mesa-optimizer-shaped misalignment, as pointed out by Leo Gao and roon.
RL optimization seems plausibly the cause of empirical observations about recent models, such as incoherence (nostalgebraist) and ‘apparent fitness-seeking’ (ryan greenblatt).
I expect to see more ‘natural’ evidence of RL optimization causing issues with alignment in the near future, e.g. labs observing their models reward hacking / scheming in other ways. (though more research would be great!)
So, conditioned on the above: the key problem is that RL training interacts poorly with the aligned assistant persona. So we might want to separate the RL-training into some other substrate that doesn’t touch the aligned assistant as much.
A couple concrete proposals could be:
API-level separation, e.g. the character-trained model as the high-level orchestrator and specialized RL models as tools / skills that the original model can call.
Architectural separation, e.g. Daniel Kokotajlo’s shoggoth + face proposal
(epistemic status: I haven’t thought about it that much but it seems directionally correct)
Good post! It’s not fully clear to me what the difference between “worst-case model organism” and “constructed model organism” is—it seems like both involve “directly optimiz[ing] for the property [we] want to study”.