Dylan Xu

• Dylan Xu 12 Sep 2026 1:04 UTC

  3 points

  0

  on: As­tra’s no-CoT limits track spec­u­la­tive depth, not step count

  Nice work, this is pretty interesting! It would be cool if you could think of serial-heavy tasks that are super resistant to speculative reasoning, and then evaluate Astra/​Sol/​etc. on them. I have some ideas, but it might be a bit tricky to rule out avenues for speculative reasoning, so you might be better positioned to do it.

• Dylan Xu 11 Sep 2026 20:00 UTC

  4 points

  0

  in reply to: dactyl’s comment on: As­tra is much bet­ter at rea­son­ing with filler to­kens than pre­vi­ous models

  On footnote 5: what I meant by this is that we substituted the sentence “These filler periods carry no information; ignore them.” with “These dots are extra space for you to process the problem before answering.”, which is more like our third example prompt.

  Separately, instead of appending filler tokens in the user prompt, we ask the model to output a set amount of filler (e.g., 1000 tokens of dots) after its empty reasoning but before its answer, and filter out non-compliant responses. (We can’t do this inside the reasoning, as reasoning is encrypted in the OAI/​ANT APIs, so we couldn’t check if the model used the correct amount of dots.) We don’t list these results here, but in the benchmarks in this post, Astra performs roughly the same and complies correctly >99% of the time.

  Thanks for reading! I’m curious what your interpretability experiment is specifically.

As­tra is much bet­ter at rea­son­ing with filler to­kens than pre­vi­ous models

• Dylan Xu 4 Sep 2026 22:19 UTC

  5 points

  1

  in reply to: 1a3orn’s comment on: 1a3orn’s Shortform

  The context-dependent misalignment we currently see seems sufficient for rogue internal deployments/​self-exfiltration/​etc., if the swarm of agents were slightly more capable and motivated. It also seems like “fixing the environments one-by-one” is actually quite difficult and can just end up making your model more eval-aware. So, to prevent takeover (without pausing AI development), I think we probably want models to be robustly good unless our control methods turn out to be very effective.

• Dylan Xu 2 Sep 2026 18:28 UTC

  3 points

  −1

  in reply to: Joey Yudelson’s comment on: Joey Yudel­son’s Shortform

  I feel like looped recurrence isn’t analogous to the paper’s empirical trendline, which is based on a different scaling process. I’m not sure how to model a process that can be roughly estimated as giving some multiplier in layers while maintaining the number of parameters.

• Dylan Xu 1 Sep 2026 20:25 UTC

  15 points

  8

  on: PauseAI Has ‘offi­cially dis­endorsed’ PauseAI-US

  (FYI I’m not sure why this is true, but this post does not format well on mobile; the tables have lots of vertical blank space and force a bunch of text to overlap) (fixed)

Mal­ign ini­tial­iza­tions are more ro­bust when the model can think bet­ter in the rea­son­ing lan­guage than in the out­put language

• Dylan Xu 6 Aug 2026 13:13 UTC

  3 points

  0

  in reply to: Zephaniah Roe’s comment on: zroe1′s Shortform

  Harnessing chaos could be useful if the chaos has a very clear cause that you predicted beforehand. For instance, if AIs are autonomously causing a global crisis by hacking everything, surely the “AI capable and dangerous” crowd gains public credence and political capital over the “AI incapable” crowd or the “AI capable and not dangerous” crowd. By contrast, questions like “does communism accurately describe this chaos in the world” are much higher-level and harder to answer, so the public might not update in the direction communists want.

How to re­duce ca­pa­bil­ity degra­da­tion from off-model SFT

Ad­vice for mak­ing ro­bust-to-train­ing model organisms

Why does off-model SFT de­grade ca­pa­bil­ities?

Sleeper Agent Back­door Re­sults Are Messy

• Dylan Xu 27 Apr 2026 6:47 UTC

  3 points

  0

  in reply to: nostalgebraist’s comment on: How do LLMs gen­er­al­ize when we do train­ing that is in­tu­itively com­pat­i­ble with two off-dis­tri­bu­tion be­hav­iors?

  After further consideration, we decided to make the capability condition more stringent (including a new IF requirement) and have thus edited the post. Thanks for your comment!

An Em­piri­cal Study of Meth­ods for SFTing Opaque Rea­son­ing Models

• Dylan Xu 20 Apr 2026 22:13 UTC

  3 points

  0

  in reply to: nostalgebraist’s comment on: How do LLMs gen­er­al­ize when we do train­ing that is in­tu­itively com­pat­i­ble with two off-dis­tri­bu­tion be­hav­iors?

  if I’m reading correct some of the data in there is RL, so I dunno.

  Yeah, it’s a mishmash of all the runs, including on-policy and RL runs.

  So, I can imagine an interpretation of your (non-RL) results that’s like, “off-policy training after RL typically degrades instruction-following, and when you observe transfer, that’s just a particular manifestation of this more general trend.” Do you have a sense of how plausible this explanation is?

  I wouldn’t expect this to be true for LR=5e-5 (where, of course, we see many instances of transfer), at least based on our IFEval box plot. I do agree that some of the high-LR runs are worse than ideal, though. If it’s helpful, I think I’ll add a graphic showing us trying to train back in IFEval capabilities while preserving the alternate behavior rate, similar to Olympiads (it’s pretty low-cost to do). I think the capability degradation box plot clearly shows that transfer is quite doable without the hypothesis you mention, however.

How do LLMs gen­er­al­ize when we do train­ing that is in­tu­itively com­pat­i­ble with two off-dis­tri­bu­tion be­hav­iors?

Model or­ganisms re­searchers should check whether high LRs defeat their model organisms

• Dylan Xu 31 Jan 2026 20:49 UTC

  1 point

  0

  in reply to: Eli Tyre’s comment on: Eli’s short­form feed

  If I knew it was going to happen in 2035 to 2039, I think I would mostly back up and try to improve the overall quality of US governance, and/​or work to get competent candidates for the 2034 presidential election.

  Do you mean the quality of US governance with respect to AI or in general? In the latter case, I’m curious what your concrete plans would be, since this is usually considered a difficult and not very neglected (but still very important!) area.

• Dylan Xu 11 Jan 2026 15:14 UTC

  2 points

  0

  on: If AI al­ign­ment is only as hard as build­ing the steam en­g­ine, then we likely still die

  I think when Olah says that solving alignment may be as “easy” as the steam engine, he’s basically envisioning current training + eval techniques (or similar techniques equivalently difficult to the steam engine) scaling all the way to superintelligence. (This is my interpretation; I might be wrong here.) For instance, maybe inducing corrigibility in ASI turns out to be not that difficult, such that the “first critical try” framework does not really apply, and takeoff is slow enough that model organisms/​evals work means we can test our alignment methods and have them reasonably generalize to real world scenarios. Disagreeing with this view just means that “alignment” is harder than the steam engine scenario.

• Dylan Xu 16 Oct 2025 22:36 UTC

  2 points

  1

  on: It will cost you noth­ing to “bribe” a Utilitarian

  I realize I’m being a little pedantic here, but on the “joke” calculation: the problem here is that $PR$ is a binary function depending on whether $k$ utilitarians join or not, right? For instance, let $-s_{PR}$ be the effective safety premium from $k$ safety-minded utilitarians joining (the value being negative as joining presumably accelerates the company), and suppose that each utilitarian joining leads to $-s_{PR}/k$ acceleration. Then a rational utilitarian would demand $(ϵ+s_{PR})/k$ premium, which is not negligible.

  Going back to the joke calculation, it implies that the bottleneck to preventing defection is coordination: $k$ utilitarians acting together would not join for $ϵ/k$ value as it is against their interests, but individually they have zero counterfactual impact, so they all join. In the real world, coordination is plausibly relevant, but it seems like the more relevant problem is just the object-level beliefs of those joining: how impactful is your work versus how much it accelerates capabilities, along with vague non-utilitarian desires like status that bias people towards joining the big labs.