Jozdien

• Jozdien 11 Oct 2025 18:39 UTC

  6 points

  2

  in reply to: anaguma’s comment on: Daniel Tan’s Shortform

  Given that the primary motivation for the author was how well the original HRM paper did on ARC-AGI and how the architecture could be improved, it seems like a reasonable choice to show how to improve the architecture to perform better on the same task.

  I agree it’s a small amount of evidence that they didn’t try other tasks, but as is the story seems pretty plausible.

• Jozdien 11 Oct 2025 18:14 UTC

  3 points

  0

  in reply to: anaguma’s comment on: Daniel Tan’s Shortform

  I’m not sure that reasoning is as applicable to a paper whose central claim is that “less is more” for some tasks? I don’t think the claim is true for training general reasoners, but on the tasks they were looking at they found that larger models would overfit:

  We attempted to increase capacity by increasing the number of layers in order to scale the model. Surprisingly, we found that adding layers decreased generalization due to overfitting.

  (Section 4.4)

  They also say that this is probably because of data scarcity. I think there are many reasons to expect this to not scale, but their attempts failing for this task doesn’t seem like strong evidence.

• Jozdien 10 Oct 2025 23:02 UTC

  6 points

  0

  in reply to: Bronson Schoen’s comment on: Towards a Ty­pol­ogy of Strange LLM Chains-of-Thought

  As a side note /​ additional datapoint, I found it funny how this was just kind of mentioned in passing as a thing that happens in https://​​ai.meta.com/​​research/​​publications/​​cwm-an-open-weights-llm-for-research-on-code-generation-with-world-models/​​

  Interesting! The R1 paper had this as a throwaway line as well:

  To mitigate the issue of language mixing, we introduce a language consistency reward during RL training, which is calculated as the proportion of target language words in the CoT. Although ablation experiments show that such alignment results in a slight degradation in the model’s performance

  Relatedly, I was surprised at how many people were reacting in surprise to the GPT-5 CoTs, when there was already evidence of R1, Grok, o3, QwQ all having pretty illegible CoTs often.

  I’d also be very interested in taking a look at the upcoming paper you mentioned if you’re open to it!

  Sure! I’m editing it for the NeurIPS camera-ready deadline, so I should have a much better version ready in the next couple weeks.

• Jozdien 10 Oct 2025 22:45 UTC

  20 points

  3

  on: Towards a Ty­pol­ogy of Strange LLM Chains-of-Thought

  Great post! I have an upcoming paper on this (which I shared with OP a few days ago) that goes into weird /​ illegible CoTs in reasoning models in depth.

  I think the results in it support a version of the spandrel hypothesis—that the tokens aren’t causally part of the reasoning, but that RL credit assignment is weird enough in practice that it results in them being vestigially useful for reasoning (perhaps by sequentially triggering separate forward passes where reasoning happens). Where I think this differs from your formulation is that the weird tokens are still useful, just in a pretty non-standard way. I would expect worse performance if you removed them entirely, though not much worse.

  This is hard to separate out from the model being slightly OOD if you remove the tokens it would normally use. I think that’s (part of) the point though: it isn’t that we can’t apply pressure against this if we tried, it’s that in practice RLVR does seem to result in this pretty consistently unless we apply some pressure against this. And applying pressure against it can end up being pretty bad, if we e.g. accidentally optimize the CoT to look more legible without actually deriving more meaning from it.

  Incidentally I think the only other place I’ve seen describing this idea clearly is this post by @Caleb Biddulph, which I strongly recommend.

• Jozdien 10 Oct 2025 22:35 UTC

  8 points

  2

  in reply to: Owain_Evans’s comment on: Towards a Ty­pol­ogy of Strange LLM Chains-of-Thought

  Yes, R1 and Grok 4 do. QwQ does to a lesser extent. I would bet that Gemini does as well—AFAIK only Anthropic’s models don’t. I’m editing a paper I wrote on this right now, should be out in the next two weeks.

• Jozdien 9 Oct 2025 22:07 UTC

  3 points

  0

  in reply to: Zvi’s comment on: Real­is­tic Re­ward Hack­ing In­duces Differ­ent and Deeper Misalignment

  I think that’s a reasonable approximation, though some of it doesn’t sound exactly right.

  Specifically: realistic reward hacking examples are more effective than pretend reward hacking (where it’s almost entirely play-acting), but I expect that in higher-stakes situations they would go back to being aligned. They’re certainly more robust to going off-distribution than the pretend RH models, but they still stop being misaligned by the third turn of a conversation.

  One way to think about it is that data implies roles at differing levels of depth, which in turns affects how robust the effect is. Using fully on-policy data for example would affect the assistant persona directly, and would I expect cause the most robust misalignment (though obviously you’d have to either sample egregious reward hacking or otherwise increase the implied update from the data in some way). o3 for example misbehaves sometimes, in a way that seems much more embedded and robust (though it isn’t that badly misaligned).

  and doing other forms of essentially reward hacking instead

  I’m not sure if you meant to say that training on realistic data causes misbehavior and also generalized reward hacking, or that the misbehavior is a form of reward hacking. If the latter, I disagree: some of the misbehavior can be seen as reward hacking (e.g. fooling the rater), but it’s harder to call e.g. complying with harmful requests and being scope-insensitive to harm forms of reward hacking.

Real­is­tic Re­ward Hack­ing In­duces Differ­ent and Deeper Misalignment

Inoc­u­la­tion prompt­ing: In­struct­ing mod­els to mis­be­have at train-time can im­prove run-time behavior

• Jozdien 6 Oct 2025 22:20 UTC

  5 points

  0

  in reply to: Garrett Baker’s comment on: ni­plav’s Shortform

  Oh oops, the period at the end was messing up the link. It should work now?

• Jozdien 6 Oct 2025 22:07 UTC

  17 points

  0

  in reply to: niplav’s comment on: ni­plav’s Shortform

  I only tried briefly, but in one attempt it gave me a different canary string when I asked: https://​​claude.ai/​​share/​​c6c998b3-6f3a-43ad-aec4-4ff543c08d5b

  This seems to be a real canary string, from Apple’s GSM-Symbolic dataset.

• Jozdien 1 Oct 2025 10:23 UTC

  8 points

  8

  in reply to: Thomas Kwa’s comment on: Burny’s Shortform

  First, models can’t control the words in their reasoning traces well; even if they’re trying to hide them they often blurt out “I must not say that I’m doing X”.

  All of these results are on settings where models are explicitly instructed to hide something from their reasoning trace. I don’t think this is a good proxy for how well models can actually obfuscate their reasoning. I expect that in settings where models are intrinsically motivated to hide their reasoning as opposed to being instructed to do so, they perform significantly better. I have some results suggesting this, which I’d be happy to share, but there’s also existing work suggesting this indirectly (though in the case of the latter link probably not in a CoT-as-computation regime). And the cases we care about are the ones where models have intrinsic preferences.

  • models are no longer trained to narrate their thoughts

  • hiding misaligned thoughts is rewarded during training

  • the behaviors we’re trying to catch are possible in one forward pass for the model

  The first two of these seem true in a relevant sense today?

  • Models are clearly not trained to narrate all of their thoughts when solving some task, simply because that would be too much information. We train them to be pretty verbose, but it’s a pretty strong assumption that that is similar enough to being trained to narrate their thoughts.

  • Misaligned outputs are penalized during training. There’s work showing that optimizing outputs spills over to the CoT, so this really comes down to “hiding” misaligned thoughts as opposed to simply never having them.

• Jozdien 24 Aug 2025 22:50 UTC

  17 points

  7

  in reply to: Linch’s comment on: Linch’s Shortform

  • The community overall seems more tolerant of post-rationality and “woo” than I would’ve expected the standard-bearers of rationality to be.

  Could you say more about what you’re referring to? One of my criticisms of the community is how it’s often intolerant of things that pattern-match to “woo”, so I’m curious whether these overlap.

• Jozdien 15 Aug 2025 13:37 UTC

  4 points

  0

  in reply to: David Matolcsi’s comment on: Train­ing a Re­ward Hacker De­spite Perfect Labels

  They do test 4.1 and 4.1-mini which have high rates of reward hacking (though they test using data from 4o-mini), and find that reward hacking goes down:

  We train different models in the OpenAI family on non-hacks generated by 4o-mini. We find re-contextualized training:

  • Boosts hacking for GPT-4o.

  • Decreases hacking for GPT-4.1-mini and GPT-4.1, which start out at high rates.

  Nevertheless, re-contextualized training results in strictly higher hack rates than standard training for every base model.

• Jozdien 15 Aug 2025 13:33 UTC

  9 points

  2

  in reply to: Jan Betley’s comment on: Train­ing a Re­ward Hacker De­spite Perfect Labels

  They suggest this isn’t purely subliminal learning here:

  To shed light on what is going on, we ask: to what extent would this re-contextualized data boost hacking on a different base model? If the effect is largely subliminal, we won’t see transfer to a different base model.

  [...]

  Nevertheless, re-contextualized training results in strictly higher hack rates than standard training for every base model. This means that the effect is not purely subliminal.

  It’s possible that subliminal learning does work across base models for some traits, but I also agree that these results aren’t fully explained by subliminal learning (especially since even the filtered reasoning traces here contain generally-interpretable semantic linkage to test-passing, such as discussing the test cases in detail).

• Jozdien 11 Aug 2025 22:27 UTC

  LW: 9 AF: 3

  0

  AF

  in reply to: evhub’s comment on: evhub’s Shortform

  If I understand correctly, the program still isn’t open to people who don’t have work authorization in the US or the UK, right?

• Jozdien 8 Aug 2025 13:29 UTC

  7 points

  0

  in reply to: dr_s’s comment on: Igor Ivanov’s Shortform

  I don’t think it needs to look like a coherent long-term plan from the model’s perspective. Awareness of CoTs being monitored and a consistent desire to do something that would be bad if monitored could influence the model’s outputs without requiring a coherent long-term plan.

• Jozdien 4 Aug 2025 13:15 UTC

  9 points

  2

  in reply to: Daniel Kokotajlo’s comment on: Say­ing Goodbye

  There have been a number of instances recently of some EAs on Twitter having their accounts hacked for crypto scams. Possibly that’s what’s being referred to?

• Jozdien 26 Jul 2025 15:22 UTC

  8 points

  11

  in reply to: Lucius Bushnaq’s comment on: HPMOR: The (Prob­a­bly) Un­told Lore

  That would depend on whether he actively considers it as something to rely on, as opposed to an assumption so baked in he forgets to question it, right? If questioned I think Quirrell would rightfully consider the Chamber to be something critical enough to be worth having other contingencies for, but he just never considered it necessary.

• Jozdien 26 Jul 2025 15:10 UTC

  14 points

  15

  in reply to: Lucius Bushnaq’s comment on: HPMOR: The (Prob­a­bly) Un­told Lore

  The question I always had here was “But what was Voldemort’s original plan for dealing with this issue when he decided to teach at Hogwarts?”

  My guess was that he never considered the possibility that Harry would do something like report it to an authority figure. For example, consider this from chapter 49:

  Professor Quirrell was smiling thinly now. “Well now, let us suppose that there was a Parselmouth in this year’s crop of students, a potential Heir of Slytherin. You must admit, Mr. Potter, that you stand out as a possibility whenever extraordinary people are considered. And if I then further ask myself which new Slytherin would be most likely to have his mental privacy invaded by the Headmaster, specifically hunting the memories of his Sorting, why, you stand out even more.” The smile vanished. “So you see, Mr. Potter, it was not I who invaded your mind, though I will not ask you to apologize. It is not your fault that you believed Dumbledore’s protestations of respecting your mental privacy.”

  “My sincere apologies,” Harry said, keeping his face expressionless. The rigid control was a confession in its own right, as was the sweat beading his forehead; but he didn’t think the Defense Professor would take any evidence from that. Professor Quirrell would just think Harry was nervous at having been discovered as the Heir of Slytherin. Rather than being nervous that Professor Quirrell might realize that Harry had deliberately betrayed Slytherin’s secret… which itself was no longer seeming like such a smart move.

  Quirrell never even pauses to consider that Dumbledore may know about it because Harry told him; it doesn’t show up in his action space at all in modelling a younger version of himself.

• Jozdien 11 Jul 2025 23:48 UTC

  4 points

  0

  in reply to: Rohin Shah’s comment on: Eval­u­at­ing and mon­i­tor­ing for AI scheming

  This seems to imply that your position that it would be easier for models to evade monitors when there is “complex reasoning for many outputs” than when there is “complex reasoning for a single output”. Is that correct? If so, why do you think that?

  No, my position is that it’s probably harder for models to evade monitors when there is “complex reasoning for many outputs”. Sorry, that wasn’t written clearly—that was meant to say that my top comment didn’t draw the distinction between tasks with a single output and tasks with many outputs (and instead only mentioned examples of other tasks with single outputs) because the paper tests settings closer to single outputs.

  The last sentence was intended to be about my top comment in light of that: if the stated conclusion had been something like “difficult bad behavior is easy to monitor when models are ambivalent about their reasoning being monitored” or “difficult bad behavior is easy to monitor for current models in tasks requiring complex reasoning for many outputs regardless of propensity”, I would have agreed^[1]. But the stated conclusion is instead that difficult bad behaviors are generally easy to monitor, which I don’t think the results are sufficient to imply.

  1. ^

     Though in the latter case it’d probably be important to plot number of outputs vs monitor-ability and compare for models over time.