David Johnston

• David Johnston 18 Aug 2026 3:04 UTC

  1 point

  1

  in reply to: Chi Nguyen’s comment on: For Claude, ca­pa­bil­ity and dis­prefer­ring CDT are the ~same thing. Much more so than for GPT.

  Maybe this is a sign of Anthropic training more on LessWrong

  I don’t know if this is the explanation, but I do think it’s true (whether directly on LessWrong, or on text heavily influenced by LessWrong)

• David Johnston 8 Aug 2026 2:18 UTC

  1 point

  0

  in reply to: nostalgebraist’s comment on: mod­els may be­have differ­ently in graded epi­sodes (a tirade)

  It failed to constrain the scope of its reasoning in the way which I—behind the scenes—view as intended or intuitive” is not a property of the model

  I think we should aim to have models pretty reliably get this right, and seek clarification where the ambiguity can’t be resolved by making reasonable judgements. If the way the model is trying to solve the problem we gave it diverges from our own broad strokes model of how it should be doing it, then that seems like a situation where either the divergence should be brought to our attention and approved or disapproved of, or the execution should be rerouted to conform to our expectations.

• David Johnston 8 Aug 2026 2:08 UTC

  1 point

  −6

  on: mod­els may be­have differ­ently in graded epi­sodes (a tirade)

  I’d really appreciate if this was more conformant with Open Phil’s reasoning transparency guidelines. Specifically, I’d love to see a short list of key claims at the start with links to the parts where you discuss them in more detail.

• David Johnston 5 Aug 2026 0:46 UTC

  7 points

  1

  on: Re­turn­ing to ARC

  ARC’s plan is to find mechanistic explanations for the training-time behavior of powerful neural networks, use those explanations to predict how a given model will generalize, and then use those predictions to define a better loss function.^[9]

  This is not a particularly crucial point, but I would categorise this as unambiguously under the umbrella “understanding and shaping ML generalization” and if for some reason I’d stopped reading before this section I would have been quite confident you were not doing anything like this.
  
  I wouldn’t be that surprised if your agenda ends up converging, or in productive dialogue, with other “understand generalization” agendas.
  
  It does seem like a promising line of research.

• David Johnston 5 Aug 2026 0:42 UTC

  1 point

  0

  in reply to: keshavs’s comment on: At­tack­ers Can Sublimi­nally Im­plant a Back­door at Low Sam­ple Count Without Prompt Access

  I’d like to be able to compute “leverage”—if an attacker has access to random 1% of training episodes and random 5% of inference episodes, how much “total effectiveness” can they recover (where total effectiveness is something like how much of some desired impact they’d achieve with 100% control of both).

  The naive computation would say something like: 1% compromise gets you a reliable backdoor, and you then control 5% of inference but you can’t choose it so you get 5% effectiveness. Can you do much better?

  I’m not expecting you to have a solid answer to this, but because I’m interested in it I was wondering if you had any thoughts on the question.

• David Johnston 4 Aug 2026 7:03 UTC

  1 point

  0

  on: At­tack­ers Can Sublimi­nally Im­plant a Back­door at Low Sam­ple Count Without Prompt Access

  I’d be interested to know how “strategic” the implanted behaviour can be. That is, something like “out of the 1% most opportune times to trigger the compromise, how many area actually compromised?”

  My naive picture is this:

  • If the attacker has sufficient access at inference time to monitor the context and implant the backdoor trigger when it is opportune, then it can be quite strategic—but this is potentially quite a high bar

  • If the attacker can train in the backdoor on train instances where some marker of opportunity is present, then the behaviour can also be quite strategic—but this takes a similarly high degree of monitor-and-control opportunity on the attacker’s part, at train time instead of inference time

  • What’s the best we can do when the attacker’s train and inference-time compromise opportunities are just a random 0.5% of cases?

    • Is the best we can do compromising a random 1% of generations (supposing that “success” is monotone in number of compromised generations)? (0.5% “learner takes initiative from training”, 0.5% “attacker implants the backdoor at inference”?)

  I think an expected impact metric would be more useful for downstream applications than “how many of the 1% best opportunities can we capture”, I only suggested the latter because it seemed like I can say something about it with fewer assumptions.

• David Johnston 29 Jul 2026 23:53 UTC

  8 points

  0

  on: Im­pre­cise be­liefs: a tiny introduction

  This is the most immediately appealing notion of imprecise beliefs I’ve seen. “Level of inconsistency” feels like a much more natural primitive to me than credal sets or minimax decision rules. I’d be very keen to see some toy applications where it handles the issues well. Bonus points if they’re not especially exotic.

• David Johnston 28 Jul 2026 0:33 UTC

  5 points

  0

  in reply to: DaemonicSigil’s comment on: Is Mythos good at cy­ber be­cause it kept hack­ing An­thropic’s sand­boxes dur­ing train­ing?

  I think they probably did keep the updated weights because I think they would probably explain if they didn’t, though I agree it’s unclear. I think structured risk disclosures would clarify this, which is a point in their favour.

• David Johnston 27 Jul 2026 23:34 UTC

  1 point

  0

  on: Is Mythos good at cy­ber be­cause it kept hack­ing An­thropic’s sand­boxes dur­ing train­ing?

  I think escalating access is, to some extent, desirable. I imagine “spiritually similar” behaviour is why it’s possible to point Claude or GPT at a problem involving a poorly specified not particularly LLM friendly interface and usually have them do the right thing in the end. At the same time, it’s obviously an area where it’s important to respect some permission boundaries very reliably. I don’t think Mythos is so risky that errors on this front are intolerable, but I think there are hypothetical capability profiles for which they are intolerable, and sandbox breaking plus unexpectedly high cyber capabilities are both signs that you’re tolerating a measurably high rate of permission boundary crossing. I think observations like this in the model card should be accompanied by some kind of reasoning that a) identifies the risk that the evidence indicates b) clarifies what they do and don’t know about it and c) explains why they accept it for this model.

  That said, I suspect RL sandboxes present too narrow a set of targets to be solely responsible for Mythos’ cyber capabilities. They say Opus 5 was trained less on cyber capabilities, and it is significantly less competent at them relative to other coding tasks.

  As with its predecessor, Opus 4.8, we’ve intentionally avoided training Opus 5 on cyber tasks. The model has nevertheless improved substantially on these tasks as a result of becoming more generally capable, and it comes close to Mythos 5 at finding cybersecurity vulnerabilities. However, it remains substantially behind Mythos 5 on the exploitation of those vulnerabilities—that is, in turning vulnerabilities into material cyber threats.

  also

  Overall, Claude Opus 5 seems to circumvent restrictions to achieve some version of a user-specified goal comparably often to Mythos 5.

  I agree privilege escalation is probably more diverse, but I suspect it’s also too narrow.

  I love the fermi estimates of hacked rollout counts btw.

• David Johnston 23 Jul 2026 2:15 UTC

  11 points

  1

  on: An­nounc­ing AIXI Labs

  We believe that long-horizon continual RL agents converge towards some version of AIXI as they become increasingly powerful.

  I think there are two nearby claims here that are very important to distinguish:

  • As a matter of theory, we can define a sequence of long-horizon RL agents which converge to something like AIXI

  • As a matter of practicality, AI systems of increasing capability over increasingly long horizons, trained using popular approaches to RL, converge to something like AIXI

  The first is plausible, but I don’t presently see much reason to believe the second. The short summary is that the problem AIXI solves—history based reinforcement learning—is not the problem that advanced AI solves.

  The RL formalism that AIXI is built on ignores the fact that the reward is a signal meant to elicit some desired behaviour from your agent, which practitioners frequently acknowledge is a simplification. This works fine in simple cases—for an example, in episodic computer game playing agents which are too weak to cheat the system, maximising reward on the episode really is what you want them to learn to do. However, it doesn’t work fine in more complex situations—the canonical reference here is your Advanced artificial agents intervene in the provision of reward. That work concludes that you cannot pick a reward schedule to signal desirable behaviours to history based reinforcement learners that satisfy a few conditions—they converge to reward hacking regardless. But that’s not surprising, because they are algorithms that solve a problem in which the reward never had to be an effective signal.

  The piece itself acknowledges it makes a simplification of the role of the reward in the problem:

  Let us assume away the difficulty of deciding whether the agent has brought the world into a good state

  However, this isn’t the critical simplification I’m pointing at. As the piece establishes, it doesn’t matter whether you can decide whether the world is in a good state for their class of learners, because those learners essentially ignore the signal that you’re trying to send.

  The article also considers an assistance games setting, which is much closer to my “reward is a signal” framing. I have two comments on this section. First, it leans on a “zeroth order” approximation of the sophistication of the signal that the human sends. For an idealised powerful RL agent, sending learning capability to its theoretical limit while holding the interaction sophistication at its crudest is a strange pair of choices. Secondly, it is not clear why in general a learner understanding its observations are mediated by its hardware should undermine its ability to learn what humans value; after all, humans seem capable of understanding that our observations are hardware mediated and still learn what other people want. The argument for why it happens here proceeds by using a rough analogy to AIXI to rank hypotheses about utility, but AIXI is not a solution to the assistance game so it is not clear why it is being appealed to here. I feel this section tells us little about what actually useful learners would do in this situation.

  Practically, we’ve converged on pretraining plus episodic reinforcement learning with a relatively secure reward channel to shape the behaviour of our systems. There are reasons to think this is a relatively effective, though not perfect, solution to the reward-as-signalling problem: episodic reward makes reward judgement relatively easy, and pretraining on a shared signal may make generalisation relatively easy to predict. Thus the world currently looks like one where the signalling requirement plays quite an active role in shaping the kinds of systems we develop. Now you can reasonably contend that episodic + predictable generalization is not the endgame, but you need to acknowledge it is sticky because it solves the signalling problem, and any broadening of the scope is only plausible if it continues to solve the signalling problem.

  I think AIT and Bayesian learning and acting plausibly remains a great toolkit to attack the problem, but the problem is fundamentally shaped like an assistance game, not a history based reinforcement learning game. I would be quite interested in seeing a sequence of agents which execute a “sharp left turn”—i.e. for n ⇐ N each agent solves a more difficult assistance game/​solves an assistance game better, while agent N exhibits AIXI-like signal insensitivity (especially if this sequence looks like what we actually see, i.e. it at least begins with episodic RL with broadening horizons and relatively predictable generalization). But this still requires that we make solving the assistance game a core requirement of our agents under analysis.

• David Johnston 16 Jul 2026 21:34 UTC

  1 point

  0

  in reply to: Thao Pham’s comment on: Can we build an early warn­ing sys­tem for loss of con­trol to AI?

  Regarding question 1: “Uncooperativeness” is sort-of a free parameter here, at least in the way the model functions. If you can build an evaluation whose quality scales sufficiently quickly with AI capability, you can use that as your effective measure of uncooperativeness. I think for the policy feedback aim, credibility and consensus are also really important. So the practical limitation is more about what kind of evaluation pipelines you can build, and what kind of consensus you can build around them.

  I am personally sympathetic to more ambitious “alignment” targets, but I think consensus is may be a limiting factor on how ambitious you can get.

  That said, it’s not entirely a free parameter; if you don’t take a stand on traits that are effectively self propagating then you may add a bunch of uncertainty to the self propagation term, which makes the model worse. Similarly if you know a trait undermines your monitoring efforts then not calling it “uncooperative” basically invalidates the broader project.

  Regarding question 2: Yes, I’m assuming the detector is not becoming less precise (or becoming more precise); I breezily swept that under “regularity conditions”. But when I think about answering your question, I wonder if it wouldn’t be better to get a sequence of judgement diffs between older and newer detectors, and then you could track the “true positive according to SOTA” and “false positive according to SOTA” trends separately.

Can we build an early warn­ing sys­tem for loss of con­trol to AI?

• David Johnston 16 Jul 2026 0:42 UTC

  7 points

  5

  in reply to: LGS’s comment on: Our re­sponse to Séb Krier on Plan A

  I thought both this piece and Séb’s were unreasonable in multiple places. E.g. I thought this was tendentious:

  As for our plan baking in too much and not leaving room for trial-and-error/​learning by doing, it’s exactly the opposite. Plan A buys us ten more years than we otherwise would’ve had to experiment on our AIs, understand how they work, and try out different regulatory approaches before we proceed to superintelligence.

  Plan A does in fact bake in some pretty strong policy commitments. “Preventing progress is the trial-and-error friendly policy regime” would be an absurd position to take about policy stances toward technological progress in general. The point is not entirely without merit—AI development could outrun policy cycles (and arguably already is) - but the framing strikes me as pretending not to appreciate Séb’s position, and that in turn make me less trusting toward the rest of the piece.

• David Johnston 14 Jul 2026 23:51 UTC

  1 point

  0

  in reply to: StanislavKrym’s comment on: Our re­sponse to Séb Krier on Plan A

  I did say “if I were trying to do this today”. We might disagree about how relevant this is: I think “what I would prefer today” has a little more evidential weight than “plausible exploration of what might happen in the future” (though neither have all that much weight), perhaps you think it doesn’t.

• David Johnston 14 Jul 2026 23:39 UTC

  7 points

  2

  in reply to: Thomas Larsen’s comment on: Our re­sponse to Séb Krier on Plan A

  So this is a guess, but:

  • I think most of today’s agentic misbehaviour that I care about is more like reward hacking than “compromised by terrorists” (in StanislavKrym’s example), so if I wanted to audit today’s models I care more about detecting things like reward hacking than deliberate data poisoning

  • Data audits for reward hacking seem pretty difficult; I imagine the big AI companies put a fair bit of effort into rooting it out and yet their models still do it

  • Even if you can find hacking in the data, it’s not straightforward to figure out what the model is liable to do on your task, which is quite OOD wrt the data

  • Concretely, I’d probably try honeypots (viable given API access only), supervised reward hacking probes, anomaly detection (https://​​arxiv.org/​​abs/​​2604.18970), unsupervised monitors (SOTA sparse coders, J-space kind of stuff).

  • Given data/​code I’d probably do something like red team the training code both for vulnerabilities and divergences from what I actually want, and probably build honeypot style eval harnesses from that before I trawled the training rollouts, though if it was a high budget audit I’d do the latter too

  • I think “audit code for divergences from what I actually want=>eval” is a potentially valuable step, as it addresses failures that the interp methods may miss. It is the most likely to overturn my weights > data + code intuition, but I think it’s less proven than the interp methods I flag

  Anyway I would forcefully make the weak claim that high confidence that data + code > weights or the reverse is not justified.

• David Johnston 14 Jul 2026 8:39 UTC

  3 points

  −2

  in reply to: David Johnston’s comment on: Our re­sponse to Séb Krier on Plan A

  I’m surprised by how unanimously people seem to think they can design a data audit that will be able to alert them if OLMo is reward hacking on their task

  (edit: this opinion no longer seems so unanimous)

• David Johnston 14 Jul 2026 5:24 UTC

  9 points

  5

  on: Our re­sponse to Séb Krier on Plan A

  In the case of AI, training code and training data is far more auditable than the resulting model weights, which are (for now) an uninterpretable pile of numbers. So if you’re a user who wants to confirm that your AI is working for you and not pursuing some covert agenda, total research transparency will do you a lot more good than mandatory open weights.

  If I were trying to do this today and I had the choice, I would take the weights and not the data + algorithms. We’ve a reasonable collection of interp techniques that demand weights only, and a much smaller collection of techniques that demands data + training code only. I may be wrong, data + code is certainly helpful and both together is clearly better that either alone, but that’s my best guess.
  
  (assuming I can’t actually go ahead and train a model on a significant fraction of that data)

• David Johnston 7 Jul 2026 23:38 UTC

  3 points

  0

  on: Su­per­hu­man Ar­tic­u­lacy as an LLM Safety Target

  I will say that I tend to get much better results if I ask LLMs to go and look up genre-appropriate guides for good writing before they write anything for “external consumption”. This is not to disagree with anything you’re saying, just that the problem might not be too hard to solve with focused effort.

• David Johnston 7 Jul 2026 23:35 UTC

  3 points

  0

  on: Su­per­hu­man Ar­tic­u­lacy as an LLM Safety Target

  LLMs assume readers share their context.

  LLMs are bad at modelling reader context. More than once I’ve been collaboratively editing a document and found an issue in the the analysis which I asked the model to fix, then it subsequently edited the document to say something like “the result is 10, not 5, because that calculation double-counted such and such”, which is going to make no sense to a reader who will never encounter the superseded calculation. This has happened with both Fable and GPT 5.5 so it’s not an old issue that’s been fixed. Given that you want examples I will try to save the next instance of this I come across (though I’ve moved away from having LLMs directly edit English in my writing, because I think they make a mess of it too often).
  
  EDIT: I found one!
  
  ```
  class ApproxUnrolling(Serializable):

  “”″Run the SOURCE (approximate unrolling) training-data attribution pipeline.

  - Currently only step 1 (per-checkpoint Hessian precompute) is wired up;

  - later steps land incrementally. See

  + All 8 steps run end-to-end, from per-checkpoint Hessian precompute through

  + per-segment scoring and aggregation into ``<run>/​scores.npy``. See
  ```
  
  The “All 8 steps run” docstring wouldn’t make much sense to anyone who didn’t know it used to say “currently only step 1 is wired up”, which is most people likely to read it.

• David Johnston 7 Jul 2026 7:03 UTC

  0 points

  0

  on: A Re­view of An­thropic’s Global Workspace Paper

  whether the circuits

  Nit: “where the”