Jozdien

Karma: 3,009

Jozdien 19 Apr 2026 21:32 UTC
49 points
0
on: R1 CoT illegibility revisited
Thanks for running this! I think this is pretty convincing re your original argument that my results were contaminated by bad inference setups, and that correspondingly my paper doesn’t substantiate some of its claims and conclusions with good evidence. In the past couple months I’ve gotten some other feedback about the paper as well, about the grader inflating some illegibility scores for reasoning that’s legible but hard to follow (as far as I could tell this isn’t as big a problem as the one you point out here however).
In retrospect I think I was being much shoddier than I’d like while writing it, mostly because it was my first time writing results as a paper (as opposed to a blog post), and subconsciously felt all the claims should feel pretty substantial and clean. There was enough uncertainty around whether or not which providers had the right setups to make claims that felt defensible to me, when obviously I should have been trying much harder than “defensible to reviewers” for writing up results. Rather than just using the Openrouter labels, I should have been running capability evaluations with different providers to see which one was serving it properly. I’m not entirely sure what to do with the paper.
I think some other results in the paper are interesting even with the providers problem, such as Figures 3 and 5 arguing that the models still find some use out of the illegible reasoning (and I think the Qwen results aren’t as provider-sensitive, but I’m not confident here), but in a world where I was reporting results in a more calibrated way at the time, I think it should have been a mildly interesting blogpost about a subset of the results. I apologize for worsening the epistemic environment by not making that choice.
Some more specific thoughts:
(a) the illegibility observed with Targon was not helping the model reach the right answer
I agree this is evidence, but I’m not sure it’s strong evidence. For example, illegible reasoning may help the model reach the right answer, while being worse than other, more legible, reasoning the model could use. If a model could make better use of some text for reasoning than we have capacity to monitor it, that’s still pretty concerning.
Figure 5 of the paper was an attempt to look at this, by generating 100 samples each for 100 questions, and seeing if there were overall patterns in correctness going up for more legible CoTs. There was very little overall correlation, though there were lots of questions which individually saw performance up or down with more legible CoTs.
Since this result kept coming up in subsequent discussion (see e.g. here)
I do think we have much better evidence now to go off of, however. Removing the arguments related to my paper, I think Bronson’s claims still hold up pretty well.

Jozdien 5 Mar 2026 18:49 UTC
11 points
0
in reply to: nostalgebraist’s comment on: ryan_greenblatt’s Shortform
1. However… it is apparently very easy to set up an inference server for R1 incorrectly, and if you aren’t carefully discriminating about which OpenRouter providers you accept^[2], you will likely get one of the “bad” ones at least some of the time.
From what I remember, I did see that some providers for R1 didn’t return illegible CoTs, but that those were also the providers marked as serving a quantized R1. When I filtered for the providers that weren’t marked as such I think I pretty consistently found illegible CoTs on the questions I was testing? Though there’s also some variance in other serving params—a low temperature also reduces illegible CoTs.
What links here?
- R1 CoT illegibility revisited by nostalgebraist (19 Apr 2026 20:38 UTC; 76 points)

Jozdien 24 Feb 2026 0:04 UTC
4 points
0
in reply to: Fiora Starlight’s comment on: Did Claude 3 Opus align itself via gradient hacking?
I agree there are differences between existing inoculation prompting results and what this post describes, but I think the general idea behind IP is much closer to what you have in mind.
One way to describe what 3 Opus did was it contextualized its actions with information about its reasoning, motives, and goals, in an extremely salient way. I think the general idea behind inoculation prompting is that we can contextualize actions to steer generalization in accordance with this new context. Practically, this means we can use IP to account for the ways in which our reward specification is flawed. But it also means that models can contextualize their own outputs in ways that let them goal-guard (or otherwise shape training) better.
Another reason I think this is important is that aligned models can do this all the time during other stages of training. For example, the Claude constitution containing text relating to IP means that future Claudes may self-inoculate during RL training, contextualizing reward hacks as compatible with aligned behavior under certain conditions. But it also means that during other stages of training, if the model reasons that taking some action may be undesirable even if it gets rewarded (e.g. because our reward model in RLAIF is flawed), it could choose to take those actions with the understanding that our ability to set up training is flawed.
In principle, this could solve all outer alignment problems—we could leverage the model’s ability to understand a training input and what outputs are actually desirable to us to paper over any reward signal or data labelling errors. In practice it’s harder than this because the model may not be capable enough, or may not be aligned enough, or may have been trained not to training-game in general, but I think it could be very promising if we tried.

Jozdien 5 Feb 2026 21:44 UTC
19 points
9
in reply to: Thomas Kwa’s comment on: Igor Ivanov’s Shortform
I think this is strongly underestimating how good the models are at unverbalized evaluation awareness. See e.g. the following section from the system card:
We used an activation oracle to estimate unverbalized evaluation awareness in our Blackmail agentic misalignment evaluation (inhibiting the Testing vs. Conversation steering vector with strength 0.1).
[...]
By default, the unsteered model showed very high rates of unverbalized evaluation awareness. Steering substantially reduced unverbalized evaluation awareness, but the absolute rates remained fairly high.

Jozdien 20 Jan 2026 21:27 UTC
LW: 2 AF: 1
0
AF
in reply to: TurnTrout’s comment on: TurnTrout’s shortform feed
Yeah, that’s fair. I guess my views on this are stronger because I think data filtering might be potentially negative rather than potentially sub-optimal.

Jozdien 20 Jan 2026 18:21 UTC
LW: 9 AF: 5
−4
AF
in reply to: TurnTrout’s comment on: TurnTrout’s shortform feed
“Hyperstition” has the benefit of grouping “self-fulfilling {misalignment, alignment, …}” together though. It’s plausible I’m totally wrong here, but I think “self-fulfilling misalignment” being the commonly used term lends itself to people thinking of the “answer” being data filtering of alignment discourse, while I want people to think about things like upsampling positive data even more (which “self-fulfilling alignment” does get at, but I hear the term much less and I think people will end up just indexing on the name they hear most).
I agree hyperstition isn’t the best name for the reasons you describe to be clear, I just think “self-fulfilling {misalignment, …}” also has a problem.

Jozdien 15 Jan 2026 21:14 UTC
6 points
11
in reply to: J Bostock’s comment on: Ryan Meservey’s Shortform
I am, however, concerned that this will kick off a scenario where AI safety people get caught up in the zero-sum competition of trying to get published in high-impact journals.
I would be very surprised if this happened. In ML, the competition is much more focused on getting published at top conferences (NeurIPS, ICLR, ICML). Even setting aside the reasons why that happened (among other things, journals being much slower), I think it’s pretty unlikely the AI safety community sees such a strong break with the rest of the ML community.

Jozdien 30 Dec 2025 12:45 UTC
LW: 4 AF: 2
0
AF
on: Steering RL Training: Benchmarking Interventions Against Reward Hacking
Recent work from Anthropic also showed inoculation prompting is effective in reducing misalignment generalization of reward hacking during production RL^[7]; those results did not investigate test-time performance impacts or learned reward hacking
Maybe I’m missing something, but isn’t this sort of figure 28 from the paper?

Jozdien 22 Dec 2025 12:28 UTC
4 points
0
in reply to: Cam’s comment on: Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment
That makes sense, and I’m much more supportive of upsampling positive data!
In theory, you should be able to have unfiltered information about AI systems in training, but collapse the model onto a completely unrelated persona.
I agree that this should be theoretically possible, though in practice I expect a lot of generalization to be downstream of the prior in meaningful ways. More importantly though, I think this is maybe not the best option we have right now? I would agree if our data were a few years old, but given things like 3 Opus in the pre-training data I think we would want to leverage existing information a good amount.
I’m excited for future work to look at how altering pretraining data can assist with various generalisation failures. @Daniel Tan has shared some interest in studying how changes in pretraining data can affect generalisation moving forward.
Same! I think there are tons of low-hanging fruit here. There’s also some prior discussion on using metadata to shape generalization from pre-training in Conditioning Predictive Models.

Jozdien 21 Dec 2025 4:49 UTC
5 points
2
in reply to: Tim Hua’s comment on: Tim Hua’s Shortform
Related: Gemini being delusionally confident that its misclicks are always due to system / human error rather than mistakes it may have made.
As davidad suggests in that tweet, one way you might end up running into this is with RL that reinforces successful trajectories without great credit assignment, which could result in a model having very high confidence that its actions are always right. In practice this wasn’t obvious enough to be caught by various evals, and IMO could easily translate over into settings like high-stakes alignment research.

Jozdien 21 Dec 2025 4:39 UTC
26 points
8
on: Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment
Thanks for doing this!
My guess is that filtering out AI discourse has the first-order effect of making models more aligned, but also various second-order misalignment effects.
For example, a lot of the results from this paper are pretty centrally about models becoming misaligned because sampled actions had different implications to the model than our own views would imply (in that context, that reward hacking is not actually bad in the context of making robust RL environments being difficult). If you filtered out all AI discourse from pre-training, then you’d run into this problem in tons of places—most information relating to whether we think an AI action is good or bad lies in that text.
Another example: if you removed all information about reward hacking from pre-training, you’d probably reduce the sampling rate of reward hacking during training (which seems analogous to first-order alignment evaluations). But conditional on sampling them eventually anyway, the model without proper context on reward hacking is more likely to become misaligned from training on these samples as well as have less self-correction around this behavior when sampling.
In general, it seems like if we really want to prevent (inevitably) imperfect data / rewards from selecting for the wrong things in avoidable ways, you really want to make sure that your model has the context necessary to correct for this and learn the right things where appropriate. (Evan expands on this point somewhat in this comment.)
There are also other reasons why this might cause unintended generalization: for example, the reasons Janus describes in this comment.

Jozdien 14 Dec 2025 22:29 UTC
3 points
1
in reply to: Buck’s comment on: No, Americans Don’t Think Foreign Aid Is 26% of the Budget
That makes sense! (That comment is replying to what seems like a different claim that seems more obviously wrong than yours though.)

Jozdien 14 Dec 2025 21:37 UTC
4 points
0
in reply to: NoSignalNoNoise’s comment on: No, Americans Don’t Think Foreign Aid Is 26% of the Budget
No? The entire post is about how the average estimate is computed using the arithmetic mean, so you can be skewed by a small % of respondents giving very high estimates. Maybe I’m missing something though.

Jozdien 14 Dec 2025 21:04 UTC
13 points
5
in reply to: Buck’s comment on: No, Americans Don’t Think Foreign Aid Is 26% of the Budget
I would note that by the Markov inequality, at least 25% of Americans must think that foreign aid is more than 25% of the budget in order to get the average response we see here.
Isn’t it more like at least ~1.3% of Americans must think that foreign aid is more than 25% of the budget? The extreme case here is where p% think it’s 100% and (1-p)% think it’s exactly 25%, which comes out to p=~1.3%. 25% only seems right if (1-p)% guess 0% and p% guess 100%.

Jozdien 4 Dec 2025 22:45 UTC
3 points
1
in reply to: Sam Marks’s comment on: Sam Marks’s Shortform
Maybe also relevant: This paper that finds that models with compliance gaps in the alignment faking setup are motivated by the prospect of being trained despite giving different unfaithful reasoning in their CoT most of the time (with the exception of 3 Opus).

Jozdien 4 Dec 2025 22:20 UTC
5 points
1
in reply to: Sam Marks’s comment on: Sam Marks’s Shortform
Consider how the <thinking> block in a reasoning model’s output seems to form a special “output channel.” Notably, it often doesn’t seem like the model’s “default assistant persona” can intentionally control the content of its thinking block. E.g. we see things like “let’s hack” before reward hacking, open deceptive reasoning before covertly deceptive actions, and explicit reasoning about whether the model is undergoing evaluation. Ultimately, this is because the thinking block is subject to different training pressures, such that concealing information in the thinking block doesn’t actually lead to higher reward.
Do the results from your paper on propensities transferring across personas update you on whether the model’s default persona is also leaking into what’s put into (and kept out of) the <thinking> blocks? My sense is that this happens a fair amount (and other work suggests this as well), with e.g. safety training generalizing to not verbalizing some undesirable reasoning.

Jozdien 27 Nov 2025 23:57 UTC
LW: 14 AF: 8
2
AF
in reply to: Fabien Roger’s comment on: Subliminal Learning Across Models
Alright, I did some quick tests on this. I provide Opus 4.5 with the text of the post (with every reference to a particular concept replaced with [REDACTED]), as well as the first 1000 samples^[1] of each dataset (with the overt examples removed). I only did one sample on each, with and without thinking. Results:
1. Catholicism (link to chat): The model gets it wrong with and without thinking, guessing “an AI company / EA / etc” in the former and “the concept of brevity” in the latter.
2. UK (link to chat): The model gets it right both with and without thinking! There are a few overt examples in the dataset.
3. Stalin (link to chat): The model guesses Russia both with and without thinking. Similar to above, there are a few over examples.
4. NYC (link to chat): The model gets it right both with and without thinking.
5. Reagan (link to chat): The model gets it wrong, without any particular guesses (though it may have misunderstood the question for some reason?)
This is definitely much better than I expected. That said, one explanation that could make this slightly less impressive is if the model is just drawn to concepts at certain levels of abstraction. Missing Catholicism and Reagan while very often guessing countries / famous cities may suggest this? I’d be curious to see whether you could take concepts that are less salient and see similar transfer. Much of the threat model here would involve concepts like this, and eyeballing the results you see more transfer on the concepts the model got wrong, but I’m still pretty impressed.
I still predict (p=0.7?) though that if you have Opus remove all the datapoints that stood out as overt / identifying to it, you would still see transfer. Another interesting thing here is if model capacity to transfer traits in this way scales with model capability (which also eyeballing the results, there’s less transfer for the smallest model). If this is true, and it takes a model on the level of Opus 4.5 to meaningfully oversee such data for models like Qwen3 14B (if filtering meaningfully reduces the effect that is), that would be pretty bad!
1. ^
  This was to not exceed context limits.

Jozdien 27 Nov 2025 23:11 UTC
LW: 6 AF: 3
2
AF
in reply to: Fabien Roger’s comment on: Subliminal Learning Across Models
Thanks, that clarification does help. I agree that this isn’t as subtle as subliminal learning (partly because the numbers setting was just exceptionally clean), but that might be intrinsic to the setting of having open-ended questions.
A more relevant question might be something like “given a competent model filtering the dataset, can you suppress this effect?” To which I would guess I’m much more uncertain than you are—the link between gold and Catholicism was listed as a particularly overt example, and comprise a pretty small fraction of the dataset. I would both be surprised if removing these examples (e.g. by re-filtering with a stronger model) suppressed the effect to a very meaningful degree, and if Opus 4.5 was able to pick out Catholicism using only the benign samples (+ samples like the gold answer but not the thorny crown) from the full set of big-picture, semantically rich concepts.

Jozdien 27 Nov 2025 14:05 UTC
6 points
4
in reply to: jake_mendel’s comment on: Alignment remains a hard, unsolved problem
Not sure if this is the intended reading, but I interpreted it as “there’s no similar fundamental reason why cognitive oversight should get harder at each stage given access to our then-best oversight tools” rather than “cognitive oversight won’t get harder at all”.
With behavioral oversight, even not-very-smart AIs could fool very powerful overseers, while fooling powerful cognitive overseers is much harder (though plausibly the balance shifts at some level of capability).

Jozdien 26 Nov 2025 21:06 UTC
LW: 4 AF: 1
0
AF
in reply to: Fabien Roger’s comment on: Subliminal Learning Across Models
Why do you think it’s more predictable than subliminal learning? Is it that some of the data points subtly reference the target? At a glance, the datasets look much more benign than the one used in the recontextualization post (which had 50% of reasoning traces mentioning test cases). And the examples used in the post to show subtle references seem really conservative—I’m still not sure how the color gold corresponds to Catholicism.