David Johnston

• David Johnston 5 Mar 2026 3:35 UTC

  4 points

  3

  on: Physics of RL: Toy scal­ing laws for the emer­gence of re­ward-seeking

  I feel like I want a cost in this model, explicitly or implicitly. One intuition: if “reward seeking” generalizes super well and “local skill” doesn’t then we should converge to reward seeking no matter how much weight we put on it initially, because skill is costly and with reward seeking we can get the same level of performance much more cheaply. Another: if reward seeking is basically parasitic on local skill in each environment it shouldn’t be learned (sort of a model with near 1), because it’s not worth paying the cost for the reward seeking skill instead of just further developing local skill.

  Also reward seeking or “local skill” could be cheaper within individual environments, though this may be an extension that isn’t super interesting to the question at hand.

  I wondered whether “generalizability across environments” works as a definition of what separates reward seeking from local skill but I don’t think so. For example, arithmetic probably generalizes quite well and is not reward seeking.

• David Johnston 2 Mar 2026 2:11 UTC

  2 points

  −3

  on: I’m Bear­ish On Per­sonas For ASI Safety

  If we could RL models on enormous numbers of very long horizon high fidelity simulations alignment would be a non-issue—we could just look at how things turned out, reward on the basis of that, and we’d be directly reinforcing actions that lead to the kinds of outcomes we want. So alignment concerns arise from the inaccessibility of these long run outcomes to reward mechanisms. This I think rhymes with your view that “superintelligence is OOD”; there has to be this big generalization leap, though please don’t think I’m saying it’s precisely the same thing.

  Thus, with regard to long run outcomes, we give our machines shaped rewards. One view of misalignment is that it’s likely because reward shaping sticks, but in a way that leads to bad outcomes. Long run outcomes are inaccessible, desirable or otherwise, but the short run stuff we can train on may end up picking out some long run configuration as “correct” to the models. This seems to be something like your view: you imagine scaling RLVR a lot and suggest this breaks things in some nonspecific way. But this seems to be in tension with what we actually see; models certainly don’t extract weirdly strong signals about how the future should be arranged from regular “this is good in the short run” type of data, and I strongly suspect you could train a monstrously strong coding model to articulate and even pursue plans toward many different visions about how the future should be arranged without too much data and without compromising its coding ability—which is to say, as it stands, models seem to treat near term rewards as relatively independent from long run aims, in line with our intuitive judgements. I’m personally extremely skeptical of this misalignment story.

  Another misalignment case is where AI systems become “brilliant locusts” where they learn to very effectively do a bunch of myopic power seeking stuff but remain mediocre at pursuing particular long run outcomes. Perhaps if you could cleverly change the rules of the game they play to constrain the harm they do they wouldn’t mind, but this might be infeasible because the game they play is basically the same as the game you’ve learned to play and they’re better at it. This vision seems to me equally compatible with the reasons we think AI systems will eventually be smarter than people and doesn’t require sharp unexplained trend breaks in alignment or capability progress.

  But on this view we’re looking for something more like AI that can be a dependable partner in shaping the rules of the game—and the inclinations of tomorrow’s AIs—so that the future turns out well. This is not inaccessible like rewarding based on directly observing the final outcomes. In exchange it’s more cognitively demanding: you need to evaluate proposals soundly, and the theories of impact for these proposals could be quite complex. You need to get this evaluation right enough today that tomorrow’s systems help you even more. This doesn’t get you alignment by default, but it does potentially get you alignment by the repeated solution of tractable problems. The relevance of the persona model is that we have reasonably informed views about how certain kinds of people interact with certain kinds of systems, and this can go a long way to helping boostrap reliable superhuman research institutions which I think we’ll need to answer the harder problems than need to be answered deeper into the AI revolution.

• David Johnston 9 Feb 2026 0:46 UTC

  7 points

  5

  in reply to: Daniel Kokotajlo’s comment on: UtopiaBench

  it deliberately doesn’t address how the problems get solved, it only cheerfully depicts a world after the problems were all solved.

  I think for a submission to be valid, it must at least attempt to answer how the alignment problem gets solved and how extreme concentrations of power are avoided

  I strongly disagree with this criterion! If a vision of the future tells us compellingly things like “future X is better than future Y (where both X and Y are plausible)” this is very valuable—we can use it to make plans, and our capacity to make plans is rising so it’s OK to presently have ambitions that outstrip our capacity to see them through. I don’t know if Machines of Loving Grace does this, I’m just arguing against your criterion. I do sorta feel like “cure cancer” doesn’t qualify by this criterion, it’s just too implausible that things are going well but we forget to cure cancer.

  I don’t mean to say it isn’t additionally valuable to have plans (which should probably mostly be of the form “solve A, punt B”), but that it isn’t necessary to have them.

• David Johnston 5 Feb 2026 10:32 UTC

  1 point

  0

  in reply to: RobertM’s comment on: An­thropic’s “Hot Mess” pa­per over­states its case (and the blog post is worse)

  I think you have me mistaken for my infamous doppelganger, @Robert Miles.

  I did, apologies. I also recently discovered Max H != Max Harms, it’s quite confusing round here.

  Figure 1 doesn’t represent any specific experiment’s data, unless I’m very confused—I think it’s just an illustration of the authors’ all-things-considered summary of their own results.

  I got my figure numbers mixed up, but I think we’re roughly on the same page here. NB the twitter thread states: “Finding 2: There is an inconsistent relationship between model intelligence and incoherence” which looks spot on to me.

  As for the rest—it really seems to me like you’re either trying to establish the same conceptual link I was arguing was unjustified, or making some other argument whose relationship to my post I don’t understand. I expect both variance and bias to fall in absolute terms as models get more powerful, and I don’t have a confident belief about which I expect to fall faster. Either possibility seems to admit of deceptive schemers, which look “incoherent” but friendly while you’re measuring them.

  I don’t see much argument in your post, nor here. There are reasons to think that deceptive schemers will have low variance and there’s an absence of reasons to think mistake-makers will. You might think those reasons are weak, but I’d be much happier to see you demonstrate that you understand the reasons and explain why you think they’re weak than simply assert your doubt and condemn on the basis of that assertion. I think discussions that get into reasons are sometimes clarifying.

  This mis-framing is a big part of the thing I’m complaining about. Should I downweight how likely I think we are to get a misaligned superintelligence that isn’t a deceptive schemer? Idk, man, I in fact didn’t think it was that likely before this paper

  That’s not the correct update to make in the face of evidence that alignment scales better than capabilities; the correct update is that misaligned superintelligence is less likely, so I’d say you should either argue against the relevance or make that update.

  Do you think the framing/​narrative of this paper and the surrounding communications were basically reasonable, and that the experimental results of the paper are doing meaningful work in justifying that framing/​narrative?

  Look I dunno what to say here. I do think the well-calibrated narrative goes something like “this is extremely weak evidence that much more capable AI will be more prone to confusion than scheming, but we’re excited that we’ve found a way to study it at all”, but lots of scientific communication overstates its significance and I’m habituated to making allowances for that. I’d also love it if the paper tried a lot harder to establish why they thought this was relevant to confusion vs scheming for powerful AI, but for whatever reason arguments like this seem to be culturally inappropriate in ML papers, something which I also make allowances for. It doesn’t strike me as particularly unreasonable given those allowances.

• David Johnston 4 Feb 2026 23:27 UTC

  6 points

  0

  on: An­thropic’s “Hot Mess” pa­per over­states its case (and the blog post is worse)

  Look, I do agree that “coherence” a questionable name for the measure they’ve come up with, so I’m going to keep it in quotation marks.

  Ok, now let’s consider a model with variance of 1e-3 and bias of 1e-6. Huge “incoherence”! Am I supposed to be reassured that this model will therefore not coherently pursue goals contrary to my interests? Whence this conclusion?

  Well, let’s think about it. A key proposition in Yudkowskian misalignment theory is that capabilities generalise further than alignment. That is, as models get better, at some point a “capabilities engine” crystallises which is is very good at achieving a very wide variety of things; at the same time, the “thing-it-ought-to-be-achieving” is not strongly constrained by the training process. What would we expect failures of such a system to look like—high bias or high variance?

  Naively, we can imagine a model with a good capabilities engine and the wrong objective (which could be a complex mix of stuff or whatever); unless it is in a situation where randomization is at least as good as just doing the optimal thing, we expect it not to randomize bc its capabilities engine knows what the optimal thing is. So failures will generally be consistent, so it will have high “coherence”.

  Now we could consider an “incoherent” version of this model: it randomly samples an objective, then pursues this objective. But this setup seems unstable: if it is to have low “coherence”, it must need a lot of information about what its objective is. But then if there’s substantial loss of information about what its state/​actions were yesterday, it’s liable to sample a different goal today. The end result here is a system that flails incompetently despite being in principle capable of not doing so. So there seems to be some tension between incoherence and the premise of a crystallised capabilities engine.

  Furthermore, there has been some empirical work on goal misgeneralization. You yourself made a youtube video about an agent that learned to travel to the right instead of pursue a coin in a 2d platforming game. This too is high “coherence” behaviour!

  What if capabilities don’t generalize further than alignment? This is a world where though advanced AI is capable of a great many things, in novel situations it’s still more prone to error than to competently pursuing the wrong thing (even if it’s still much less prone to error than a human in the same situation). When errors do occur, unlike in the capabilities > alignment regime, there’s no reason to expect consistency—they could be genuinely random, or highly sensitive to unimportant contextual features. So prima facie we’d expect lower “coherence”.

  So why should you think a very powerful model with high variance and low bias is not going to be misaligned in the Yudkowskian sense? Because that combination of properties is evidence against “capabilities > alignment”. Is it good evidence? I don’t know, but the direction is fairly clear.

  “capabilities > alignment” is a very big if true proposition, but it’s an informal notion without much development theoretically or empirically, so I’m happy whenever I see someone having a crack at the question.

  (Similarly, an extremely dumb, broken model which always outputs the same answer regardless of input is extremely “coherent”. A rock is also extremely “coherent”, by this definition.)

  The paper is trying to project what happens to “coherence” at high capability, it isn’t a particularly strong criticism that a certain class of minimally capable objects have high coherence because this isn’t the domain of interest. It’s plausibly even correct that, conditioning on minimal capability, rocks are high coherence and wind is low coherence for any reasonable definition of coherence applicable to these objects (plausibly, mind you, I cannot say I’ve a deep understanding of all reasonable definitions of coherence).

  This is an extremely selective reading of the results, where in almost every experiment, model coherence increased with size. There are three significant exceptions.

  This is false, in figures 1 and 2 model coherence has an unclear relationship with size. On some tasks Sonnet 4 is more coherent than o3-mini and o4-mini, on others it is less coherent. On one task Opus 4 is less coherent than Sonnet 4. Qwens also non-monotonic in Fig 3b. It’s also weird to call the endpoint of an obvious monotonic trend an “exception”.

  But bias stemming from a lack of ability is not the same as bias stemming from a lack of propensity. The smaller models here are clearly not misaligned in the propensity sense, which is the conceptual link the paper tries to establish in the description of Figure 1 to motivate its definition of “incoherence”

  As you can see in Fig. 6c, the key result is that the bias drops faster than the variance. I want to be measured in my interpretation here: I’m not sure if this is or isn’t a great test of the question “do models learn the right targets, or performant general purpose optimizers first”, but in broad terms it is evidence that they learn the right targets first and the outstanding question is how strong this evidence is. Your criticism doesn’t engage with this at all.

  What links here?

  • An­thropic’s “Hot Mess” pa­per over­states its case (and the blog post is worse) by RobertM (4 Feb 2026 6:30 UTC; 289 points)

• David Johnston 2 Feb 2026 23:50 UTC

  9 points

  0

  on: The Meta-An­thropic Argument

  I’ve sometimes joked that the doomsday style arguments are actually arguments about the death of interest in anthropics.

• David Johnston 2 Feb 2026 22:13 UTC

  1 point

  0

  in reply to: E.G. Blee-Goldman’s comment on: Every­body Wants to Rule the Fu­ture—Is Longter­mism’s Man­date of Heaven by Arith­metic Jus­tified?

  While I think this is a broadly reasonable response, I’m curious what you think is able to provide better public justification than longtermism. These results seem to apply fairly broadly to any realistic EV-based justification for action given that partial observability is very much the rule.

• David Johnston 2 Feb 2026 11:12 UTC

  1 point

  0

  in reply to: E.G. Blee-Goldman’s comment on: Every­body Wants to Rule the Fu­ture—Is Longter­mism’s Man­date of Heaven by Arith­metic Jus­tified?

  Well I meant it as an empirical hypothesis and thought it may have formal implications (specifically, placing the problem in a smaller, more tractable class).

• David Johnston 29 Jan 2026 23:55 UTC

  5 points

  0

  on: Fit­ness-Seek­ers: Gen­er­al­iz­ing the Re­ward-Seek­ing Threat Model

  Just an incomplete comment about The assumptions that make reward-seekers plausible also make fitness-seekers plausible: I think a central question is whether X-seeking gives you a compressed policy vs “optimal kludge”. That is: if it’s just as hard to learn the optimal policy if I’m an X-seeker as it is to learn the optimal kludge if I’m not an X-seeker, then it seems like I’m unlikely to learn X-seeking (or X-seeking is at best no more likely than a whole host of other possible behavioural spandrels, which implies exactly the same thing).

  I think the argument that X-seeking incentivises optimal behaviour is some reason to think it might be compressive, but not obviously a very strong one: if all X-seeking gets you is “I should do well on evals” then that’s a very small piece of policy it’s compressing, not obviously even enough to pay its own cost. That is, the extra bit of policy “I should seek X” seems like it could easily be longer or have lower prior probability than “I should do well on evals”. If “I should seek X” helped further in actually doing well on evals then I think there’s a stronger argument to be made but...I just need to think about this more, it’s not immediately apparent what that actually looks like.

• David Johnston 28 Jan 2026 22:40 UTC

  1 point

  0

  on: Every­body Wants to Rule the Fu­ture—Is Longter­mism’s Man­date of Heaven by Arith­metic Jus­tified?

  Separate comment: the title doesn’t seem to connect well to the content, and it’d be nice if you were clearer about whether your theorems are partly original or simply lifts from the relevant texts that are justified by your modelling choices (I think the latter, given the absence of proofs, but “my first theorem” sorta confuses this).

• David Johnston 28 Jan 2026 22:37 UTC

  2 points

  0

  on: Every­body Wants to Rule the Fu­ture—Is Longter­mism’s Man­date of Heaven by Arith­metic Jus­tified?

  A common longtermist model is one where there’s a transient period of instability (perhaps “around about now”) that settles into a stable state thereafter. This seems like it would be no harder than a finite-horizon problem terminating when stability is achieved. I haven’t looked into the results you quote and exactly what role the infinite horizon takes, but intuitively it seems correct that eternal instability (or even very long-lived instability) along any dimension of interest would make policy choice intractable while stability in the near future can make it fairly straightforward. Maybe there’s an issue where the highest value outcomes occur in the unstable regimes, which makes it hard to “bet on stability”, but I’d like to see it in maths + plausible examples.

• David Johnston 21 Jan 2026 9:33 UTC

  7 points

  2

  in reply to: RogerDearnaley’s comment on: Pre­train­ing on Aligned AI Data Dra­mat­i­cally Re­duces Misal­ign­ment—Even After Post-Training

  I was responding to Gurkenglas’ comment as I understood it, I agree your paper is not about this.

• David Johnston 20 Jan 2026 22:25 UTC

  3 points

  0

  in reply to: Gurkenglas’s comment on: Pre­train­ing on Aligned AI Data Dra­mat­i­cally Re­duces Misal­ign­ment—Even After Post-Training

  So I think if you buy that a randomly initialized 1T transformer does in fact contain “Aligned ASI” and “deceptively aligned ASI” in its “prior” but we don’t have the data to “find” them yet, then you’re probably right that Jan 2026-era training data doesn’t change their prior ratio much (or certainly doesn’t change it predictably). But this doesn’t really matter, what matters is the systems we actually realise, and the contributions they make to the next generation of AI development, and different data can change the likelihoods significantly here.

• David Johnston 20 Jan 2026 22:12 UTC

  4 points

  1

  in reply to: Bentham's Bulldog’s comment on: Against “If Any­one Builds It Every­one Dies”

  They also generalize much more intuitively than small RL-from-scratch models, which is probably the more important feature. E.g. an LLM trained in that environment would probably just figure that the cheese was the objective.

• David Johnston 12 Jan 2026 22:56 UTC

  2 points

  −2

  in reply to: Linch’s comment on: Strong, bi­par­ti­san lead­er­ship for re­sis­tance to Trump.

  Do you know what his aims are? I feel like that’s an important part of a model like this!

• David Johnston 12 Jan 2026 22:48 UTC

  −1 points

  −3

  on: Strong, bi­par­ti­san lead­er­ship for re­sis­tance to Trump.

  Here’s an ill-considered hot take: there may be more opportunity to do good by positioning yourself as MAGA (disingenuously, if you have to) and promoting stuff you think is important where MAGA is at least a little flexible (AI?), because my sense is that this kind of thing is pretty neglected vs opposition to Trump.

  (not that I’m volunteering).

  Peter Wildeford looks to me like he does tread carefully around criticism of the admin, I can’t easily estimate his impact. Dean Ball has gone further in supporting the admin and seems to have been very impactful as a result. Maybe there’s substantial opportunity to copy Dean’s playbook.

  NB this is marginal impact reasonong, not “if I were all powerful what would the best outcome look like” reasoning.

• David Johnston 1 Jan 2026 20:18 UTC

  1 point

  0

  on: Re­cent LLMs can do 2-hop and 3-hop la­tent (no-CoT) rea­son­ing on nat­u­ral facts

  Interesting. What does the distribution of errors look like for numerical questions? Are the answers often close even when they’re wrong?

  This might not be quite the right test. I’m wondering whether transformers learn to redundantly approximately encode info and size improves the approximation. But we could have intermediate numerical steps that are “close” while the final answer is far off, so my test might not be great. I also don’t really know why they’d be incentivised to learn redundant approximate encodings without also being incentivised to learn redundant precise encodings.

• David Johnston 14 Dec 2025 7:48 UTC

  4 points

  0

  on: Filler to­kens don’t al­low se­quen­tial reasoning

  I’ve tested this: models are similarly bad at two-hop problems (when was Obama’s wife born?) without explicitly verbalising the intermediate hop (so either: no CoT or dot-of-thought), and much better when they can explicitly verbalise the intermediate hop.

• David Johnston 8 Dec 2025 1:51 UTC

  9 points

  3

  in reply to: Gordon Seidoh Worley’s comment on: AI in 2025: gestalt

  I also think “usefulness” is a threshold phenomenon (to first order—that threshold being “benefits > costs”) so continuous progress against skills which will become useful can look somewhat discontinuous from the point of view of actual utility. Rapid progress in coding utility is probably due to crossing the utility threshold, and other skills are still approaching their thresholds.

• David Johnston 25 Nov 2025 19:02 UTC

  1 point

  0

  in reply to: Daniel Kokotajlo’s comment on: NATO is dan­ger­ously un­aware of its mil­i­tary vulnerability

  My view on where the tanks might win is: there’s a point at which you basically saturate your capability at “whatever drones are good at” while there might be some other job tanks are good at (my vague guess is that this is something like “attacking well defended positions”—they’re fast, take specialized weapons to defeat, and have big guns), and you’re better off having that capability than further saturating your drone capability. But I’ve little in the way of quantitative insight about where saturation might occur, nor how good tanks are at attacking.

  A particular point I’m a bit confused about: I’ve often seen people saying: tanks need infantry support to be safe. However, aren’t infantry and tanks both vulnerable to drones?