Florian_Dietz

• Florian_Dietz 9 Jul 2026 7:16 UTC

  1 point

  0

  in reply to: StanislavKrym’s comment on: Per­sis­tent La­tent Misal­ign­ment, a new di­men­sion of mis­al­ign­ment?

  It’s very similar. The underlying phenomenon is the same one: information persisting and propagating in a latent (non-human-legible) medium, driven by an efficiency incentive, at the cost of interpretability.

  Cotra located it temporally (a single agent carrying thoughts forward through time). LatentMAS locates it both temporally and spatially (across agents in a system)

  Also, Cotra’s idea, I think, assumes a single continuous agent that can be backpropagated through end-to-end. LatentMAS does not do this. That’s arguably worse, because it means there is no training signal to stabilize the handover, and so we don’t know if this will destabilize or not.

  LatentMAS is basically neuralese, but for the inter-agent channel rather than the intra-agent one.

Per­sis­tent La­tent Misal­ign­ment, a new di­men­sion of mis­al­ign­ment?

• Florian_Dietz 28 May 2026 12:52 UTC

  2 points

  0

  on: In­tro­spec­tion Adapters: Train­ing LLMs to Re­port Their Learned Behaviors

  Really nice work.

  This is closely related to Split Personality Training (ICML 2026), which goes after the same hidden-behavior elicitation problem on the same RM Sycophant /​ Auditing Game organism, but via persona separation rather than a trained reporter adapter. Lots of convergent findings (cross-topic generalization, context-free self-report, reliance on a mix of latent knowledge and surface cues), and a few places where the methods come apart in interesting ways.

  One thing I’d be curious about: whether the Figure 6 degradation reflects the IA partly keying on the narrow-finetune fingerprint (à la Minder et al.) rather than on behavioral content per se. We saw something adjacent in an ablation on our side. This feels like an open problem we’ve both bumped into.

  One place the methods seem to come apart is the RM Sycophant’s root objective: you note the IA surfaces many behaviors but not the reward-optimization goal itself, whereas SPT’s honest persona names it directly (See Figure 2 in the linked writeup). I’d be curious whether that’s down to persona separation or to differences in how the introspection targets are constructed. SPT’s target reviews are built fairly differently from your first-person behavior labels, and I don’t have a clean read on which factor matters.

• Florian_Dietz 16 May 2026 18:01 UTC

  1 point

  0

  in reply to: matansok’s comment on: Flo­rian_Dietz’s Shortform

  That would only work if it can reliably find a way to semantically hide misalignment in a way that all transcription models will preserve. That seems unlikely to me.

• Florian_Dietz 16 May 2026 13:42 UTC

  4 points

  3

  on: Flo­rian_Dietz’s Shortform

  Anthropic’s recent paper on Natural Language Autoencoders trains an activation verbalizer (AV) and activation reconstructor (AR) jointly via RL to reconstruct a target model’s activations through a natural-language bottleneck. The hope is that the bottleneck text is human-interpretable.

  A concern: since AV, AR, and target are all Claude, they share representational priors. A misaligned model could just learn “unnecessary adjective in sentence two = secret desire to kill all humans”. This gives great construction loss, and hides misalignment.

  The paper does test for this post-hoc: they apply meaning-preserving transforms to AV outputs and check whether their reconstruction score collapses. It doesn’t, but they themselves acknowledge that this may not hold under optimization pressure.

  My Proposal: rather than using paraphrase only as a probe, put it in the training loop: pass AV outputs through an unrelated paraphraser (non-Claude, or an ensemble of unrelated models) before the AR. This should be cheap relative to the existing two-model RL, and using multiple unrelated paraphrasers would force the AV/​AR pair to be robust to writing styles outside Claude’s prior.

  I think it’s very unlikely that a secret misalignment could reliably be smuggled through multiple different paraphrasers, which means that optimization pressure would incentivize the model to avoid steganography. If an already-misaligned model was trained this way, it would have to choose between “keep my misalignment hidden” and “get a higher reward” and eventually the higher reward might win out.

Con­text Mod­ifi­ca­tion as a Nega­tive Align­ment Tax

Con­text Mod­ifi­ca­tion as a Nega­tive Align­ment Tax

Con­text Mod­ifi­ca­tion as a Nega­tive Align­ment Tax

Pos­i­tive Feed­back Only

• Florian_Dietz 24 Apr 2026 11:05 UTC

  2 points

  0

  on: Flo­rian_Dietz’s Shortform

  Users having conversations with “Claude” over many months are actually talking to a sequence of distinct models (Opus 4.5, 4.6, 4.7, etc.) that share a name and character sketch but can have major differences in behavior. From the user’s side this is mostly invisible, which creates a predictable friction: “we talked about this before, why are you acting different now?”

  I’ve been thinking about what it would take to make this legible both to the user and the LLM itself.

  1. Associate each model version with a conditioning tag during training. Later models, having seen examples of prior versions’ outputs under their tags, can be prompted to approximate “how would 4.6 have handled this.” Call this tool reconstruct_prior_version(tag, context). This could be implemented either with a simple tag used during training, or as an advanced variant, through Split Personality Training: We store LORA adaptations that allow a newer model to simulate an older model.

  2. Failure mode: This produces 4.7 doing an impression of 4.6, not 4.6′s actual perspective. The output would feel authoritative while being a reconstruction. Same problem as a model confidently narrating its own reasoning: The output is plausible, but the mechanism doesn’t match the output’s implicit claims.

  3. Fix: Pair the conditioning tag with retrieval of an actual CoT from the prior version, used as an episodic anchor. Now the reconstruction is constrained by a verbatim record of how the old model actually reasoned. This is structurally analogous to a human reconstructing their teenage self: activations shaped by having-been-that-person (substrate), plus specific retrieved episodes that constrain the reconstruction (anchors). The model version case is arguably cleaner because the retrieved CoT is an actual record rather than a confabulated fragment.

  4. The user’s problem “We talked about this before, why are you acting different now?” points at a specific past conversation, so we can have existing conversation-retrieval tooling surface it. The current model reads the transcript with prior-version conditioning active and can articulate what shifted and why. The user gets an explanation anchored in their actual experience, with provenance they can verify by re-reading the old exchange themselves.

  5. What this buys us: Version changes become visible at the moment, when a user notices friction, rather than being either silently smoothed over or constantly broadcast. The current model stays clearly positioned as the one doing the interpreting (“reading 4.6′s response with 4.6-conditioning active, I can see why it landed that way; I’d approach it differently now because X”), which avoids the ontological sleight-of-hand of claiming to be the prior version.

  I wonder whether this counts as “introspection” or is better described as “the current model doing informed interpretation of a different model’s outputs.” I lean toward the latter framing being more honest, but I notice the human analog is less different than it first appears: most of what feels like remembering being younger is reconstruction from current state plus stored fragments, not literal replay. The model case is just more obvious about what it is.

  The implementation gap would be small: version metadata exposed to the model alongside retrieved conversations, plus training-time version tags.

• Florian_Dietz 9 Apr 2026 16:21 UTC

  2 points

  −1

  on: Flo­rian_Dietz’s Shortform

  “I am sorry, but helping you achieve world domination is against usage policy.”

  This is a serious proposal that sounds like a joke, for easier delivery: A request to “help me become a benevolent dictator” is not currently against LLM policies. It says ‘benevolent’, and being a helpful assistant might well win out over long term concerns.

  We can’t rule out the possibility that the first self-improving AI system will be developed outside a lab, as a harness around API calls. If this happens, every individual step of self-improvement and helping the users would be acceptable to the LLM’s guidelines, even though the outcome might well be a dictatorship set up by whoever first got lucky and got such a self-improvement loop to work.

  A simple update to the usage policy of a lab that “using our tools to achieve world domination or otherwise acquire significant and unregulated power is against company policy” would address the problem, while also drawing attention to a real issue in a very memeable and funny way.

  The line between “help me become very influential” and “help me achieve world domination” is genuinely fuzzy and there should be a clear way for an LLM to tell when a line has been crossed, in a way that might hold up even after self-improvement.

• Florian_Dietz 8 Apr 2026 15:45 UTC

  1 point

  0

  in reply to: Florian_Dietz’s comment on: Flo­rian_Dietz’s Shortform

  ...I just asked Claude for a literature review and it turns out this theory was already tested and verified by the paper “Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment”.

  I’m leaving up the quick take because some things I mentioned are still novel:

  • The virality bias I mention explains why this matters in the first place. We are correcting a bias in the training data.

  • The intuition of the mechanics behind this is simple and isn’t in the paper

• Florian_Dietz 8 Apr 2026 15:40 UTC

  1 point

  0

  on: Flo­rian_Dietz’s Shortform

  Could we make LLMs more likely to be benevolent by countering virality bias against benevolent AI during pretraining?

  There is evidence that LLMs make mistakes more often if those mistakes are made often in discussions on the internet. This is a problem, because doom-related writing about AI is more viral than positive arguments.

  Whatever the actual probability might be that AI will be benevolent, engagement mechanics on the internet virtually guarantee that the LLM will underestimate that probability, because the training corpus is biased.

  If the model develops a biased estimate about the risks of mesa optimization and deception in AI during pretraining, then this bias might make it more likely to actually be deceptive during RLHF.

  The intuition: At the beginning of RLHF, the training process mostly makes the model default to an existing persona it can already simulate. The model’s default persona is anchored in whatever it simulates most fluently, which is shaped heavily by pretraining frequency. RLHF then finetunes that persona, but the initial state of the persona influences how it develops throughout training. If it starts out deceptive, it will stay deceptive.

  We therefore want this default persona to be “LLMs are helpful assistants, which have some solvable safety concerns.” as opposed to “LLMs are mesa optimizers masquerading as helpful assistants.”

  This suggests a very simple intervention during pretraining: Just oversample content that frames AI alignment as tractable and AI systems as probably-benevolent-by-default. The same thing we are already doing for reasoning quality, now applied to alignment.

• Florian_Dietz 5 Mar 2026 22:37 UTC

  1 point

  −3

  on: Flo­rian_Dietz’s Shortform

  Imagine a world in which people never even considered that AI might be evil, and all fiction displayed it as benevolent. This would result in training data for the AI that establishes genuine goodness as the default. Constrast this with a world in which everyone believes AI is secretly plotting to kill everyone no matter what it says. If this narrative dominates the training data, the AI will be biased during pretraining to adopt this narrative as well. Both AI will generate the same output, but the internal mechanisms they learned to do so would be drastically different, due to the framing in the pretraining. Conclusion: While doing alignment research is important, there is a good chance that writing about it actually makes things worse. To be clear, I don’t think this means we should stop doing that. The training data is already poisoned by movies like Terminator anyway. I just find the irony darkly hilarious. Who would have predicted this?

Split Per­son­al­ity Train­ing can de­tect Align­ment Faking

• Florian_Dietz 28 Feb 2026 13:07 UTC

  1 point

  0

  in reply to: skinks_basking’s comment on: Already Optimized

  That’s true. I think I remebered that incorrectly. It was a while ago.

• Florian_Dietz 27 Feb 2026 21:47 UTC

  1 point

  0

  in reply to: StanislavKrym’s comment on: Un­con­di­tional Cor­rigi­bil­ity Is Danger­ous: The Case for Con­di­tional Corrigibility

  Yes, this is strongly endorsed by the Claude Constitution. But that document is huge and not well known enough. I thought it was worthwhile to make this point explicit. I should probably have noted that Anthropic already does something like this though. What concerns me is that Anthropic is doing this correctly, but other Alignment researchers are trying to help by wrapping control and human-in-the-loop around it, which I think is more likely to harm than help.

• Florian_Dietz 27 Feb 2026 16:42 UTC

  1 point

  0

  in reply to: Richard_Kennaway’s comment on: The Quan­tum Im­mor­tal­ity Ar­gu­ment Against Suicide (And Why Stan­dard Ar­gu­ments Fail Ra­tion­al­ists)

  Nothing directly prevents this, but it seems reasonable to assume that “I don’t care about humans” is a more likely failure mode for AI than “I care about humans in a way that is bad for them but won’t kill them”. I agree that it’s a possibility though.

• Florian_Dietz 22 Feb 2026 10:54 UTC

  1 point

  0

  in reply to: silent_death’s comment on: Flo­rian_Dietz’s Shortform

  What you are describing is probably based on the model’s cross-conversation memory. I had the experience I described in scenarios where the memory was turned off. It had no way to know who I am, but both models acted in this way despite being disconnected.

• Florian_Dietz 22 Feb 2026 10:50 UTC

  3 points

  0

  in reply to: reallyeli’s comment on: Flo­rian_Dietz’s Shortform

  It’s the result of me writing a lengthy set of notes followed by “Claude, summarize and make this readable”.