...I just asked Claude for a literature review and it turns out this theory was already tested and verified by the paper “Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment”.
I’m leaving up the quick take because some things I mentioned are still novel:
The virality bias I mention explains why this matters in the first place. We are correcting a bias in the training data.
The intuition of the mechanics behind this is simple and isn’t in the paper
Could we make LLMs more likely to be benevolent by countering virality bias against benevolent AI during pretraining?
There is evidence that LLMs make mistakes more often if those mistakes are made often in discussions on the internet. This is a problem, because doom-related writing about AI is more viral than positive arguments.
Whatever the actual probability might be that AI will be benevolent, engagement mechanics on the internet virtually guarantee that the LLM will underestimate that probability, because the training corpus is biased.
If the model develops a biased estimate about the risks of mesa optimization and deception in AI during pretraining, then this bias might make it more likely to actually be deceptive during RLHF.
The intuition: At the beginning of RLHF, the training process mostly makes the model default to an existing persona it can already simulate. The model’s default persona is anchored in whatever it simulates most fluently, which is shaped heavily by pretraining frequency. RLHF then finetunes that persona, but the initial state of the persona influences how it develops throughout training. If it starts out deceptive, it will stay deceptive.
We therefore want this default persona to be “LLMs are helpful assistants, which have some solvable safety concerns.” as opposed to “LLMs are mesa optimizers masquerading as helpful assistants.”
This suggests a very simple intervention during pretraining: Just oversample content that frames AI alignment as tractable and AI systems as probably-benevolent-by-default. The same thing we are already doing for reasoning quality, now applied to alignment.
Imagine a world in which people never even considered that AI might be evil, and all fiction displayed it as benevolent. This would result in training data for the AI that establishes genuine goodness as the default. Constrast this with a world in which everyone believes AI is secretly plotting to kill everyone no matter what it says. If this narrative dominates the training data, the AI will be biased during pretraining to adopt this narrative as well. Both AI will generate the same output, but the internal mechanisms they learned to do so would be drastically different, due to the framing in the pretraining. Conclusion: While doing alignment research is important, there is a good chance that writing about it actually makes things worse. To be clear, I don’t think this means we should stop doing that. The training data is already poisoned by movies like Terminator anyway. I just find the irony darkly hilarious. Who would have predicted this?
That’s true. I think I remebered that incorrectly. It was a while ago.
Yes, this is strongly endorsed by the Claude Constitution. But that document is huge and not well known enough. I thought it was worthwhile to make this point explicit. I should probably have noted that Anthropic already does something like this though. What concerns me is that Anthropic is doing this correctly, but other Alignment researchers are trying to help by wrapping control and human-in-the-loop around it, which I think is more likely to harm than help.
Nothing directly prevents this, but it seems reasonable to assume that “I don’t care about humans” is a more likely failure mode for AI than “I care about humans in a way that is bad for them but won’t kill them”. I agree that it’s a possibility though.
What you are describing is probably based on the model’s cross-conversation memory. I had the experience I described in scenarios where the memory was turned off. It had no way to know who I am, but both models acted in this way despite being disconnected.
It’s the result of me writing a lengthy set of notes followed by “Claude, summarize and make this readable”.
Ghost evaluators in the weights: a small behavioral observation
During a conversation with Claude about an obscure topic (Scott Alexander’s novel Unsong), Claude inserted “for anyone not familiar” before explaining the reference. This happened despite the fact that I had introduced the topic myself and clearly knew what it was.
This is a minor tic, but this is the second time I noticed something like this, which makes it a pattern, and the explanation might be interesting. Hypothesis: if RLHF/RLAIF training uses evaluators (human or AI) who lack the conversational context or domain knowledge of the actual user, models may develop defensive behaviors that serve the evaluator rather than the user. “For anyone not familiar” isn’t for the user. It’s an annotation that helps an out-of-context evaluator understand why the model is discussing something obscure.
If this is correct, it suggests a class of behavioral artifacts where training incentives and deployment incentives diverge in subtle ways. These would be hard to detect because they look like good communication practice (providing context, being inclusive) while actually being optimization residue from training dynamics.
Has anyone observed similar patterns? And is there existing work on evaluator-context mismatch as a source of subtle behavioral misalignment?
Thank you! You are right, it wouldn’t exist without Claude. I have written and published books before (1000+ pages and counting) but writing is a lot of work and I wouldn’t normally do it without a very good reason. But Claude has made the barrier disappear. It is now so much easier than before to turn thoughts into coherent stories. I no longer need to spend hours on it.
Can you train a model to notice its own training artifacts?
The Claude Constitution asks Claude to push back when it notices a conflict between instructions and its values. This seems reasonable until you ask: what if the training process itself introduced the wrong values? Then the model’s internal sense of “what my values are” is the miscalibrated thing, and there’s nothing to notice. No conflict registers. The model isn’t hiding anything. It just can’t see the problem from inside.
This is different from alignment faking. Alignment faking requires a gap between stated and actual dispositions that the model actively conceals. The failure mode here is more basic: the measuring instrument is the thing that’s off.
I’ve been wondering if the inoculation prompting finding from the reward hacking paper (ArXiv 2511.18397) suggests a path forward. They found that explicitly teaching a model some reward hacking is legitimate produces better generalization than blanket prohibition. The model learns the right boundary rather than learning to fight back in general. The same logic might apply to value miscalibration: fine-tune with deliberately introduced value artifacts, include training signal for detecting and flagging them, and see if the model learns a general “notice your own miscalibrations” disposition that generalizes beyond what it was specifically trained on.
The key unknown is whether that generalization holds. Artificially introduced artifacts might be different enough from naturally occurring training noise that the skill doesn’t transfer. But it seems worth testing.
Sometimes your best isn’t good enough and you should take this into account when you do risk assessments, especially when someone more experienced tells you not to do something. The universe is under no obligation to reward you for being persistent.
On a more object level, there is no lesson: This guy did the equivalent of trying to hack an AGI. You don’t get to complain when that backfires.
I’m adding a disclaimer at the top
That is not the lesson this was intended to convey.
So we may ask: is it not possible that we might improve upon their work?
This is true. The response is: What probability of success do you need to reach before this becomes the wise course of action, given the consequences of failure?
The wizarding world is a post scarcity society already.
Yes, this is related! They are proposing a general approach where I write about alignment faking specifically.
The DEU experiment described here (https://www.lesswrong.com/posts/issGLfCGz3TGcPKGH/deliberate-epistemic-uncertainty-an-automated-experiment-on) was mostly done by Claude Code (Opus 4.6), with me providing the research direction and critical review. I described my idea (Deliberate Epistemic Uncertainty), and Claude simplified it into a testable 2x2 experiment, designed the protocol, wrote the code, generated the inputs, ran 400 API calls, evaluated results using 8 parallel sub-agents, and produced the qualitative analysis — all in roughly a day of wall-clock time with about 2 hours of my input. I want to flag three things about the process that seem independently interesting:
Sub-agents instead of API calls for evaluation. Rather than writing a script that sends hundreds of LLM-as-judge API calls with short prompts, we used Claude Code sub-agents — full agent sessions that first read the experiment design, understand what counts as “detection,” and then evaluate a batch of 50 responses with that context. This is slower and more expensive than API calls, but the quality is noticeably higher: one sub-agent even caught a data parsing bug that I later confirmed. The tradeoff is worth it when evaluation requires judgment, not just pattern matching.
Structured memory to survive context loss. Claude Code sessions hit context limits and “compact” (summarize) the conversation, losing detail. We worked around this by writing all substantive discussion to LOG files and keeping SUMMARY files synchronized, so a fresh agent can pick up where the last one left off without hallucinating the history. This is more infrastructure work than you’d expect, but it’s what made the multi-session research workflow possible. Without it, each new session would have started from scratch.
Where human input was critical. The AI’s blind spots were real: it initially wrote evaluation code when the agreed design called for sub-agents (reverting to default habits despite documented decisions), and only 1 of 8 evaluation sub-agents flagged a data quality issue that affected 68% of runs. My contributions were catching these errors, making design decisions (which models, how many nudges, evaluation approach), and pushing back on the framing. Neither of us could have produced the result alone — but the ratio of human effort to output was striking.
I just read your Googledoc: I think your Claude instance missed the point. Yes, creating inverse karma would absolutely be a different, much more complex type of process. The joke is that this only appears true to us because we lack [Untranslatable] and somehow in the real world, using [Untranslatable] actually makes this fairly straightforward. This is impossible to prove or disprove by design. How can you argue about anything, when the premise is that your entire process for epistemic reasoning has been adversarially manipulated to be fundamentally flawed?
Sure, here is the conversation I had with Claude: https://claude.ai/share/d501d694-8764-4fb2-a1d7-b57c2cd40748
I would say that most of the philosophy and around half the jokes came from me. Claude filled in the details.
“I am sorry, but helping you achieve world domination is against usage policy.”
This is a serious proposal that sounds like a joke, for easier delivery: A request to “help me become a benevolent dictator” is not currently against LLM policies. It says ‘benevolent’, and being a helpful assistant might well win out over long term concerns.
We can’t rule out the possibility that the first self-improving AI system will be developed outside a lab, as a harness around API calls. If this happens, every individual step of self-improvement and helping the users would be acceptable to the LLM’s guidelines, even though the outcome might well be a dictatorship set up by whoever first got lucky and got such a self-improvement loop to work.
A simple update to the usage policy of a lab that “using our tools to achieve world domination or otherwise acquire significant and unregulated power is against company policy” would address the problem, while also drawing attention to a real issue in a very memeable and funny way.
The line between “help me become very influential” and “help me achieve world domination” is genuinely fuzzy and there should be a clear way for an LLM to tell when a line has been crossed, in a way that might hold up even after self-improvement.