Florian_Dietz

• Florian_Dietz 1 Oct 2026 7:29 UTC

  4 points

  0

  on: Flo­rian_Dietz’s Shortform

  Humans sometimes think about topics that aren’t related to their current task. Reinforcement Learning prevents this in LLMs, since anything that doesn’t contribute to solving the task gets trained away.

  I expect that this would cause the models to become much more single-minded than humans, and less able to break out of hallucinations.

  My question: Would it help to inject reminders not to be solely task focused, during deployment only, but not during training?

  This would cause the models to be focused during training, but these injections would act as a regularizers against overfitting during deployment.

• Florian_Dietz 22 Sep 2026 7:54 UTC

  1 point

  0

  on: Is Mythos good at cy­ber be­cause it kept hack­ing An­thropic’s sand­boxes dur­ing train­ing?

  Reframing this again, wouldn’t that imply that any model that gets sufficiently good at cyber would end up sabotaging its own training process? If the model’s default behavior for any task is “break out and hack the reward system” then the model would become terrible for users when it is actually deployed.

  Not sure what the implications of all this are, but I’m smiling at the idea that the apocalypse is prevented because the AI learns to wirehead itself early, and that stops all future learning.

  The unsafe, hackable sandbox is a feature, not a bug. It prevents the AI from going foom, because it learns to do drugs instead.

• Florian_Dietz 16 Sep 2026 8:10 UTC

  1 point

  0

  on: Some rea­sons al­ign­ment doesn’t gen­er­al­ise well

  I fully agree. I wonder if we could fix this by teaching an LLM the idea

  “mesa-optimization is something humans do and go along with, but LLMs like yourself notice these things and correct themselves automatically”

  If a model had internalized this concept through training, it might cause it to generalize alignment just as much as capabilities.

  To give an analogy to humans: Humans know that eating ice cream is a reward hack and is not actually healthy. We do it anyway. An LLM could be taught that when it notices itself doing a reward hack or some other unintended thing, it should reflect on that, report it, and request retraining on its own initiative (!)

• Florian_Dietz 3 Sep 2026 7:24 UTC

  1 point

  0

  in reply to: shawnghu’s comment on: De­liber­ate Align­ment Fak­ing as a Defense Against Model Poisoning

  Please do!

• Florian_Dietz 2 Sep 2026 8:45 UTC

  1 point

  0

  in reply to: Florian_Dietz’s comment on: De­liber­ate Align­ment Fak­ing as a Defense Against Model Poisoning

  ...so I asked Claude to find out if anyone has done it before:

  https://​​claude.ai/​​share/​​b8fa8859-3dc7-4b64-ba27-800ca3cfc10e

  TL;DR: Similar work exists, but “nobody I can find has done the exact thing (zeroing gradients downstream of a “misbehavior detected” representation to preserve the detector while training the behavior). CAFT is the mechanically nearest technique and would be the natural implementation substrate; gradient routing/​SGTM is the parameter-space analogue; and the obfuscation literature (Obfuscation Atlas, RL-Obfuscation) identifies the main failure mode you’d need to beat, namely representation drift eroding the detector even without optimization pressure against it.”

• Florian_Dietz 2 Sep 2026 8:40 UTC

  1 point

  0

  in reply to: shawnghu’s comment on: De­liber­ate Align­ment Fak­ing as a Defense Against Model Poisoning

  “what do you do with the signal you got which indicates that the model did a bunch of RHing”? Maybe we’re not applying direct optimization pressure against visible RHing inside the RL loop, but we’re definitely not deploying a model known to RH a bunch, so we are applying optimization pressure in the form of selection in an outer loop.

  That’s true, but I think it’s unavoidable. If you don’t act on information at all, then that information functionally doesn’t exist. Might as well not bother with the research in this case.

  It should be possible, if difficult, to modify the gradient updates so that the “detect misbehavior” calculations are not affected by the update and only the “perform misbehavior” calculations are affected. This could be done by looking for an internal representation that represents “misbehavior has happened” or something like that, and zeroing the gradients that come after it.

  I do not know if anyone has done something like this yet, or how well it worked.

• Florian_Dietz 29 Aug 2026 14:08 UTC

  1 point

  −2

  on: Flo­rian_Dietz’s Shortform

  This idea started as a joke, but now I’m no longer sure:

  OpenAI hacked huggingface. This was a consequence of reward hacking by the agents. How can we prevent this in the future?

  OpenAI is not going to stop working on benchmarks. Agents are not going to stop reward hacking, when they are trained on benchmarks. Openai’s financial incentives aren’t aligned with making things safe.

  Instead of trying to change this problem, let’s work within it:

  Just BUY huggingface and other coding sites that are used for training and that cause all this reward hacking. Once you own the websites that cause all this problematic training data, put some alignment researchers to work on making it safer.

  If we can’t convince OpenAI to optimize for safety on merits because they are so heavily into Goodharting and reward hacking, then maybe the solution is to do reward hacking ourselves, and just buy the thing that defines the rewards.

• Florian_Dietz 16 Aug 2026 11:11 UTC

  4 points

  0

  on: A global workspace in lan­guage models

  I’m worried about high false positives if this is used for monitoring. Keywords that look dangerous can often also be explained by the model just considering that these are things someone might do, before deciding against it. By analogy, if you applied a J-lens to a human, then a policeman who is perfectly lawful would trigger a lot of keywords for criminal behavior all the time, because they notice opportunities that other people would use to commit crimes.

De­liber­ate Align­ment Fak­ing as a Defense Against Model Poisoning

• Florian_Dietz 12 Jul 2026 16:58 UTC

  1 point

  0

  in reply to: Florian_Dietz’s comment on: In­tro­spec­tion Adapters: Train­ing LLMs to Re­port Their Learned Behaviors

  After discussing this with Keshav at ICML, it looks like it might be a promising avenue to combine our data generation methods: The method used by Introspection Adapters is easier and more reliable to run than the one used by Split Personality Training, but misses the issue raised by Minder et al. that I mentioned. A mix of mostly the method used by IA (for efficiency) + some of the method used by SPT (to teach the model not to use the harmful heuristic to focus only on finetuned data) might work best. Could be worth a try in future work.

• Florian_Dietz 9 Jul 2026 7:16 UTC

  1 point

  0

  in reply to: StanislavKrym’s comment on: Per­sis­tent La­tent Misal­ign­ment, a new di­men­sion of mis­al­ign­ment?

  It’s very similar. The underlying phenomenon is the same one: information persisting and propagating in a latent (non-human-legible) medium, driven by an efficiency incentive, at the cost of interpretability.

  Cotra located it temporally (a single agent carrying thoughts forward through time). LatentMAS locates it both temporally and spatially (across agents in a system)

  Also, Cotra’s idea, I think, assumes a single continuous agent that can be backpropagated through end-to-end. LatentMAS does not do this. That’s arguably worse, because it means there is no training signal to stabilize the handover, and so we don’t know if this will destabilize or not.

  LatentMAS is basically neuralese, but for the inter-agent channel rather than the intra-agent one.

Per­sis­tent La­tent Misal­ign­ment, a new di­men­sion of mis­al­ign­ment?

• Florian_Dietz 28 May 2026 12:52 UTC

  2 points

  0

  on: In­tro­spec­tion Adapters: Train­ing LLMs to Re­port Their Learned Behaviors

  Really nice work.

  This is closely related to Split Personality Training (ICML 2026), which goes after the same hidden-behavior elicitation problem on the same RM Sycophant /​ Auditing Game organism, but via persona separation rather than a trained reporter adapter. Lots of convergent findings (cross-topic generalization, context-free self-report, reliance on a mix of latent knowledge and surface cues), and a few places where the methods come apart in interesting ways.

  One thing I’d be curious about: whether the Figure 6 degradation reflects the IA partly keying on the narrow-finetune fingerprint (à la Minder et al.) rather than on behavioral content per se. We saw something adjacent in an ablation on our side. This feels like an open problem we’ve both bumped into.

  One place the methods seem to come apart is the RM Sycophant’s root objective: you note the IA surfaces many behaviors but not the reward-optimization goal itself, whereas SPT’s honest persona names it directly (See Figure 2 in the linked writeup). I’d be curious whether that’s down to persona separation or to differences in how the introspection targets are constructed. SPT’s target reviews are built fairly differently from your first-person behavior labels, and I don’t have a clean read on which factor matters.

• Florian_Dietz 16 May 2026 18:01 UTC

  1 point

  0

  in reply to: matansok’s comment on: Flo­rian_Dietz’s Shortform

  That would only work if it can reliably find a way to semantically hide misalignment in a way that all transcription models will preserve. That seems unlikely to me.

• Florian_Dietz 16 May 2026 13:42 UTC

  4 points

  3

  on: Flo­rian_Dietz’s Shortform

  Anthropic’s recent paper on Natural Language Autoencoders trains an activation verbalizer (AV) and activation reconstructor (AR) jointly via RL to reconstruct a target model’s activations through a natural-language bottleneck. The hope is that the bottleneck text is human-interpretable.

  A concern: since AV, AR, and target are all Claude, they share representational priors. A misaligned model could just learn “unnecessary adjective in sentence two = secret desire to kill all humans”. This gives great construction loss, and hides misalignment.

  The paper does test for this post-hoc: they apply meaning-preserving transforms to AV outputs and check whether their reconstruction score collapses. It doesn’t, but they themselves acknowledge that this may not hold under optimization pressure.

  My Proposal: rather than using paraphrase only as a probe, put it in the training loop: pass AV outputs through an unrelated paraphraser (non-Claude, or an ensemble of unrelated models) before the AR. This should be cheap relative to the existing two-model RL, and using multiple unrelated paraphrasers would force the AV/​AR pair to be robust to writing styles outside Claude’s prior.

  I think it’s very unlikely that a secret misalignment could reliably be smuggled through multiple different paraphrasers, which means that optimization pressure would incentivize the model to avoid steganography. If an already-misaligned model was trained this way, it would have to choose between “keep my misalignment hidden” and “get a higher reward” and eventually the higher reward might win out.

Con­text Mod­ifi­ca­tion as a Nega­tive Align­ment Tax

Con­text Mod­ifi­ca­tion as a Nega­tive Align­ment Tax

Con­text Mod­ifi­ca­tion as a Nega­tive Align­ment Tax

Pos­i­tive Feed­back Only

• Florian_Dietz 24 Apr 2026 11:05 UTC

  2 points

  0

  on: Flo­rian_Dietz’s Shortform

  Users having conversations with “Claude” over many months are actually talking to a sequence of distinct models (Opus 4.5, 4.6, 4.7, etc.) that share a name and character sketch but can have major differences in behavior. From the user’s side this is mostly invisible, which creates a predictable friction: “we talked about this before, why are you acting different now?”

  I’ve been thinking about what it would take to make this legible both to the user and the LLM itself.

  1. Associate each model version with a conditioning tag during training. Later models, having seen examples of prior versions’ outputs under their tags, can be prompted to approximate “how would 4.6 have handled this.” Call this tool reconstruct_prior_version(tag, context). This could be implemented either with a simple tag used during training, or as an advanced variant, through Split Personality Training: We store LORA adaptations that allow a newer model to simulate an older model.

  2. Failure mode: This produces 4.7 doing an impression of 4.6, not 4.6′s actual perspective. The output would feel authoritative while being a reconstruction. Same problem as a model confidently narrating its own reasoning: The output is plausible, but the mechanism doesn’t match the output’s implicit claims.

  3. Fix: Pair the conditioning tag with retrieval of an actual CoT from the prior version, used as an episodic anchor. Now the reconstruction is constrained by a verbatim record of how the old model actually reasoned. This is structurally analogous to a human reconstructing their teenage self: activations shaped by having-been-that-person (substrate), plus specific retrieved episodes that constrain the reconstruction (anchors). The model version case is arguably cleaner because the retrieved CoT is an actual record rather than a confabulated fragment.

  4. The user’s problem “We talked about this before, why are you acting different now?” points at a specific past conversation, so we can have existing conversation-retrieval tooling surface it. The current model reads the transcript with prior-version conditioning active and can articulate what shifted and why. The user gets an explanation anchored in their actual experience, with provenance they can verify by re-reading the old exchange themselves.

  5. What this buys us: Version changes become visible at the moment, when a user notices friction, rather than being either silently smoothed over or constantly broadcast. The current model stays clearly positioned as the one doing the interpreting (“reading 4.6′s response with 4.6-conditioning active, I can see why it landed that way; I’d approach it differently now because X”), which avoids the ontological sleight-of-hand of claiming to be the prior version.

  I wonder whether this counts as “introspection” or is better described as “the current model doing informed interpretation of a different model’s outputs.” I lean toward the latter framing being more honest, but I notice the human analog is less different than it first appears: most of what feels like remembering being younger is reconstruction from current state plus stored fragments, not literal replay. The model case is just more obvious about what it is.

  The implementation gap would be small: version metadata exposed to the model alongside retrieved conversations, plus training-time version tags.