Chloe Li

• Chloe Li 24 May 2026 22:59 UTC

  3 points

  1

  in reply to: Julian Minder’s comment on: Syn­thetic Per­sona Pre­train­ing: Align­ment from To­ken Zero

  Nice, glad to hear my comment was useful!

  FWIW, I think it is quite interesting that token zero seems to do ~as well as msm, and this is a useful thing to know!

  Tho I think it is perhaps too strong to say zero token is better than msm or vice versa from the current results—from fig11, it seems like the difference is mainly coming from the PAP eval performance. But on PAP, the baseline attack success rate is 0%, and so is zero token (so this eval is already saturated to start with). MSM and filtering both apparently make the baseline worse and raise this to 6%? I find this pretty confusing and am unsure if this is just noise. The max metric quotes this 6% number for MSM and concludes it’s worse on this basis, but again I’m not sure how strongly one should index on this. If the conclusion hinges on the max a lot, there should prob be an error bar on the max (eg error bars on PAP performance in this case). Cool work nevertheless!

• Chloe Li 21 May 2026 22:40 UTC

  3 points

  1

  on: Syn­thetic Per­sona Pre­train­ing: Align­ment from To­ken Zero

  Are there error bars to the plot? Is the difference between midtraining and token zero in fig1 statistically significant?

  How are you doing filtering? I’m pretty confused by why filtering is worse than the baseline in fig1 - is this just noise?

  Both results suggest that the model benefits from learning harmful content alongside the moral commentary on it, rather than being shielded from harmful content entirely.

  Could just be that filtering is very inaccurate and missing a bunch of harmful content?

  Some ablations that would be cool:

  • does combining midtraining and token zero do better than doing more midtraining/​tokenzero?

  • does token zero only work if you append reflections on harmful documents? If it’s 10M benign docs only, does it no longer work?

  Edit: from fig 11 it seems like the difference between midtraining and token zero is not statistically sig, neither is filtering and baseline.

• Chloe Li 6 May 2026 22:02 UTC

  3 points

  0

  in reply to: Daniel Kokotajlo’s comment on: Model Spec Mid­train­ing: Im­prov­ing How Align­ment Train­ing Generalizes

  MSM by default happens on the base model before any chat finetuning, so llama doesn’t know it is llama at that point since it’s not been instruction-tuned yet. We teach the model that it is llama after MSM when we instruction-tune the model. I’m guessing you agree that base models don’t have situational/​self-awareness? Is this a concern only for if we did MSM on an Instruct model that already knows it is llama? (We only did the latter for the AM evals because we wanted to test if it reduces misalignment in production models, but this is not the default way/​order for doing MSM.)

  Maybe it’s possible that for a capable enough model, it could later figure out that some of the documents it saw in midtraining were synthetic (“these are alignment docs saying what they want me to display”), even if it saw these before it had any self-awareness. But things like making docs more realistic/​diverse should help

Model Spec Mid­train­ing: Im­prov­ing How Align­ment Train­ing Generalizes

• Chloe Li 11 Dec 2025 20:28 UTC

  3 points

  0

  in reply to: Sam Marks’s comment on: Sam Marks’s Shortform

  • If we ablate (a)—i.e. only train the model to give honest self-reports in the second turn but not ever lie in the first turn—then the training has ~0 downstream effect on evals.

    • Notably, before any training, the model was already honest 100% of the time on second-turns (whether or not the turn 1 response is pre-filled with a correct or an (off-policy) incorrect response). So this is not a setting where the model would have originally lied.

    • (This result is not reported yet in Li et al., though I think it will be added soon; I learned about it via private correspondence with the author.)

  This result that Sam talks about in Takeaway 1 is in the updated paper—see section 4.1 for details! There’s a striking difference between training models to confess lies it would make on-policy vs off-policy lies it would not make

ARENA 5.0 - Call for Applicants

ARENA 4.0 Im­pact Report

• Chloe Li 6 Jul 2024 22:26 UTC

  5 points

  0

  in reply to: GMM’s comment on: AI Align­ment Re­search Eng­ineer Ac­cel­er­a­tor (ARENA): Call for ap­pli­cants v4.0

  It’s a fast-growing and important field right now—there is an urgency to make progress on eval, and a rapid increase in both technical safety eval roles at AI labs and governance roles. This need and capacity for safety evals make eval skills valuable for people who want to contribute to safety now. There are many methods that have been developed and relevant engineering skills to improve, but also a lot of minefields for producing false or misleading results. We thought the latter is an especially important reason for a good curriculum to exist

AI Align­ment Re­search Eng­ineer Ac­cel­er­a­tor (ARENA): Call for ap­pli­cants v4.0

• Chloe Li 12 Dec 2023 0:27 UTC

  1 point

  0

  on: Lin­ear en­cod­ing of char­ac­ter-level in­for­ma­tion in GPT-J to­ken embeddings

  We show that linear probes can retrieve character-level information from embeddings and we perform interventional experiments to show that this information is used by the model to carry out character-level tasks.

  These two links need permission to be accessed.