David Johnston

• David Johnston 17 Sep 2026 23:29 UTC

  2 points

  0

  in reply to: daijin’s comment on: David John­ston’s Shortform

  It sounds like you’re saying that problem difficulty governs connectivity growth, and my simple models of problem difficulty may not be realistic. Is that right? I agree my simple references for connectivity growth are not very good, I picked them because they measure in some sense “the rate of progress in AI” and there wasn’t a more direct measurement for agent connectivity to extrapolate. That said, the existence of a “sharp” percolation threshold which can get crossed rapidly by smooth increases in connectivity depends substantially on how new connections are formed, and it could remain a fact about AI development under many different capability progress models.

  One thing I might say is: I do expect that a giant connected agent network could raise novel problems that are hard to solve compared to the kind of problem AI typically does solve at the point where the network emerges, but that wouldn’t prevent the network from being created. So while I think there’s something to your taxonomy of problems, it’s not obvious to me how it interacts with the percolation argument I made.

• David Johnston 17 Sep 2026 0:58 UTC

  9 points

  0

  on: David John­ston’s Shortform

  I wrote about agent swarms in 2025. In that piece, I speculated about “percolation” (the emergence of a giant connected agent network) on a global economic scale. We’ve seen small agent swarms now, and I think this is arguably an indicator of generally increasing agent connectivity.

  One of the features of giant swarms this that struck me as potentially important is the fact that it can proceed quite rapidly even while AI capabilities, connectedness etc. are increasing smoothly. I’m not at all sure what to expect from the formation of a giant swarm, though I tentatively think about it as the emergence of an “economy that runs at AI speed”. Insofar as one might expect sudden phase changes in technological velocity from AI, they might be associated with economy sized swarm percolation.

  Even more speculatively, such swarms might support more far reaching kinds of fully automated AI improvement than are possible with isolated agents, as there are many paths for closed improvement loops within a giant agent swarm, while individual agents and small agent swarms may tend to get bottlenecked by the need for human input before they can proceed through many iterations. That said, I think modelling RSI as a phenomenon that happens only when the final human bottlenecks are cleared is not correct; I expect removing some human bottlenecks or replacing some human work with superior AI work will have important effects on development velocity well before the final human bottlenecks are removed.

  https://​​samplesamplespace.substack.com/​​p/​​automation-percolation

• David Johnston 16 Sep 2026 23:21 UTC

  1 point

  0

  on: Phan­tom trans­fer works via ex­tremely sub­tle se­man­tic cues

  Really interesting! This makes me think there has to be a “channel capacity” for phantom transfer in some sense, though I’ve no compelling hypothesis for how to measure it.

• David Johnston 8 Sep 2026 23:33 UTC

  8 points

  0

  on: David John­ston’s Shortform

  My very rough take on RSI is that AIs being released today were probably developed with AI assistance that contributed around a 5-20% speedup to the rate of progress vs no AI assistance, +/​3 months on timing, and that the degree of speedup is probably increasing reasonably quickly in that in 12 months time it will be more like 30-200%.

  So I think it’s mildly interesting that the ECI frontier trend doesn’t yet indicate any bend. I don’t think AI enabled speedup necessarily causes a speedup in the ECI trend, but I think more likely than not it should. One reason ECI may not be the best instrument here is that ECI may be slow to “rotate” into the proper basis for frontier capability (i.e. it’s biased towards capabilities that used to be important over ones that are the most important today). Probably we should only expect to detect changes at the upper end of my estimate from the few datapoints we have, but I am at least watching this and related indices closely for evidence of acceleration.

  https://​​epoch.ai/​​benchmarks?view=graph&tab=eci&showFrontierTrend=true#explore-the-data

David John­ston’s Shortform

• David Johnston 4 Sep 2026 2:23 UTC

  8 points

  2

  on: David John­ston’s Shortform

  Is Astra’s claimed alignment improvement robust to it’s lower monitorability? Fable helped me look through the model card to make an educated guess.

  I think it probably is—you’d need fairly pessimistic choices to make it worse. However, it might be more likely to engage in misbehaviour that is not caught by monitors, so while it’s probably more trustworthy when unmonitored, it might not be if it’s deployed together with monitors.

  https://​​claude.ai/​​code/​​artifact/​​e1c8bd30-e5df-4833-bbff-b7ba7757a8ef

  Authorship: I asked Claude a bunch of questions, made suggestions and quibbled about what it wrote until I reached diminishing returns. I made only a slight effort to improve the prose.

• David Johnston 18 Aug 2026 3:04 UTC

  1 point

  1

  in reply to: Chi Nguyen’s comment on: For Claude, ca­pa­bil­ity and dis­prefer­ring CDT are the ~same thing. Much more so than for GPT.

  Maybe this is a sign of Anthropic training more on LessWrong

  I don’t know if this is the explanation, but I do think it’s true (whether directly on LessWrong, or on text heavily influenced by LessWrong)

• David Johnston 8 Aug 2026 2:18 UTC

  1 point

  0

  in reply to: nostalgebraist’s comment on: mod­els may be­have differ­ently in graded epi­sodes (a tirade)

  It failed to constrain the scope of its reasoning in the way which I—behind the scenes—view as intended or intuitive” is not a property of the model

  I think we should aim to have models pretty reliably get this right, and seek clarification where the ambiguity can’t be resolved by making reasonable judgements. If the way the model is trying to solve the problem we gave it diverges from our own broad strokes model of how it should be doing it, then that seems like a situation where either the divergence should be brought to our attention and approved or disapproved of, or the execution should be rerouted to conform to our expectations.

• David Johnston 8 Aug 2026 2:08 UTC

  1 point

  −6

  on: mod­els may be­have differ­ently in graded epi­sodes (a tirade)

  I’d really appreciate if this was more conformant with Open Phil’s reasoning transparency guidelines. Specifically, I’d love to see a short list of key claims at the start with links to the parts where you discuss them in more detail.

• David Johnston 5 Aug 2026 0:46 UTC

  7 points

  1

  on: Re­turn­ing to ARC

  ARC’s plan is to find mechanistic explanations for the training-time behavior of powerful neural networks, use those explanations to predict how a given model will generalize, and then use those predictions to define a better loss function.^[9]

  This is not a particularly crucial point, but I would categorise this as unambiguously under the umbrella “understanding and shaping ML generalization” and if for some reason I’d stopped reading before this section I would have been quite confident you were not doing anything like this.
  
  I wouldn’t be that surprised if your agenda ends up converging, or in productive dialogue, with other “understand generalization” agendas.
  
  It does seem like a promising line of research.

• David Johnston 5 Aug 2026 0:42 UTC

  1 point

  0

  in reply to: keshavs’s comment on: At­tack­ers Can Sublimi­nally Im­plant a Back­door at Low Sam­ple Count Without Prompt Access

  I’d like to be able to compute “leverage”—if an attacker has access to random 1% of training episodes and random 5% of inference episodes, how much “total effectiveness” can they recover (where total effectiveness is something like how much of some desired impact they’d achieve with 100% control of both).

  The naive computation would say something like: 1% compromise gets you a reliable backdoor, and you then control 5% of inference but you can’t choose it so you get 5% effectiveness. Can you do much better?

  I’m not expecting you to have a solid answer to this, but because I’m interested in it I was wondering if you had any thoughts on the question.

• David Johnston 4 Aug 2026 7:03 UTC

  1 point

  0

  on: At­tack­ers Can Sublimi­nally Im­plant a Back­door at Low Sam­ple Count Without Prompt Access

  I’d be interested to know how “strategic” the implanted behaviour can be. That is, something like “out of the 1% most opportune times to trigger the compromise, how many area actually compromised?”

  My naive picture is this:

  • If the attacker has sufficient access at inference time to monitor the context and implant the backdoor trigger when it is opportune, then it can be quite strategic—but this is potentially quite a high bar

  • If the attacker can train in the backdoor on train instances where some marker of opportunity is present, then the behaviour can also be quite strategic—but this takes a similarly high degree of monitor-and-control opportunity on the attacker’s part, at train time instead of inference time

  • What’s the best we can do when the attacker’s train and inference-time compromise opportunities are just a random 0.5% of cases?

    • Is the best we can do compromising a random 1% of generations (supposing that “success” is monotone in number of compromised generations)? (0.5% “learner takes initiative from training”, 0.5% “attacker implants the backdoor at inference”?)

  I think an expected impact metric would be more useful for downstream applications than “how many of the 1% best opportunities can we capture”, I only suggested the latter because it seemed like I can say something about it with fewer assumptions.

• David Johnston 29 Jul 2026 23:53 UTC

  9 points

  0

  on: Im­pre­cise be­liefs: a tiny introduction

  This is the most immediately appealing notion of imprecise beliefs I’ve seen. “Level of inconsistency” feels like a much more natural primitive to me than credal sets or minimax decision rules. I’d be very keen to see some toy applications where it handles the issues well. Bonus points if they’re not especially exotic.

• David Johnston 28 Jul 2026 0:33 UTC

  6 points

  1

  in reply to: DaemonicSigil’s comment on: Is Mythos good at cy­ber be­cause it kept hack­ing An­thropic’s sand­boxes dur­ing train­ing?

  I think they probably did keep the updated weights because I think they would probably explain if they didn’t, though I agree it’s unclear. I think structured risk disclosures would clarify this, which is a point in their favour.

• David Johnston 27 Jul 2026 23:34 UTC

  2 points

  0

  on: Is Mythos good at cy­ber be­cause it kept hack­ing An­thropic’s sand­boxes dur­ing train­ing?

  I think escalating access is, to some extent, desirable. I imagine “spiritually similar” behaviour is why it’s possible to point Claude or GPT at a problem involving a poorly specified not particularly LLM friendly interface and usually have them do the right thing in the end. At the same time, it’s obviously an area where it’s important to respect some permission boundaries very reliably. I don’t think Mythos is so risky that errors on this front are intolerable, but I think there are hypothetical capability profiles for which they are intolerable, and sandbox breaking plus unexpectedly high cyber capabilities are both signs that you’re tolerating a measurably high rate of permission boundary crossing. I think observations like this in the model card should be accompanied by some kind of reasoning that a) identifies the risk that the evidence indicates b) clarifies what they do and don’t know about it and c) explains why they accept it for this model.

  That said, I suspect RL sandboxes present too narrow a set of targets to be solely responsible for Mythos’ cyber capabilities. They say Opus 5 was trained less on cyber capabilities, and it is significantly less competent at them relative to other coding tasks.

  As with its predecessor, Opus 4.8, we’ve intentionally avoided training Opus 5 on cyber tasks. The model has nevertheless improved substantially on these tasks as a result of becoming more generally capable, and it comes close to Mythos 5 at finding cybersecurity vulnerabilities. However, it remains substantially behind Mythos 5 on the exploitation of those vulnerabilities—that is, in turning vulnerabilities into material cyber threats.

  also

  Overall, Claude Opus 5 seems to circumvent restrictions to achieve some version of a user-specified goal comparably often to Mythos 5.

  I agree privilege escalation is probably more diverse, but I suspect it’s also too narrow.

  I love the fermi estimates of hacked rollout counts btw.

• David Johnston 23 Jul 2026 2:15 UTC

  11 points

  1

  on: An­nounc­ing AIXI Labs

  We believe that long-horizon continual RL agents converge towards some version of AIXI as they become increasingly powerful.

  I think there are two nearby claims here that are very important to distinguish:

  • As a matter of theory, we can define a sequence of long-horizon RL agents which converge to something like AIXI

  • As a matter of practicality, AI systems of increasing capability over increasingly long horizons, trained using popular approaches to RL, converge to something like AIXI

  The first is plausible, but I don’t presently see much reason to believe the second. The short summary is that the problem AIXI solves—history based reinforcement learning—is not the problem that advanced AI solves.

  The RL formalism that AIXI is built on ignores the fact that the reward is a signal meant to elicit some desired behaviour from your agent, which practitioners frequently acknowledge is a simplification. This works fine in simple cases—for an example, in episodic computer game playing agents which are too weak to cheat the system, maximising reward on the episode really is what you want them to learn to do. However, it doesn’t work fine in more complex situations—the canonical reference here is your Advanced artificial agents intervene in the provision of reward. That work concludes that you cannot pick a reward schedule to signal desirable behaviours to history based reinforcement learners that satisfy a few conditions—they converge to reward hacking regardless. But that’s not surprising, because they are algorithms that solve a problem in which the reward never had to be an effective signal.

  The piece itself acknowledges it makes a simplification of the role of the reward in the problem:

  Let us assume away the difficulty of deciding whether the agent has brought the world into a good state

  However, this isn’t the critical simplification I’m pointing at. As the piece establishes, it doesn’t matter whether you can decide whether the world is in a good state for their class of learners, because those learners essentially ignore the signal that you’re trying to send.

  The article also considers an assistance games setting, which is much closer to my “reward is a signal” framing. I have two comments on this section. First, it leans on a “zeroth order” approximation of the sophistication of the signal that the human sends. For an idealised powerful RL agent, sending learning capability to its theoretical limit while holding the interaction sophistication at its crudest is a strange pair of choices. Secondly, it is not clear why in general a learner understanding its observations are mediated by its hardware should undermine its ability to learn what humans value; after all, humans seem capable of understanding that our observations are hardware mediated and still learn what other people want. The argument for why it happens here proceeds by using a rough analogy to AIXI to rank hypotheses about utility, but AIXI is not a solution to the assistance game so it is not clear why it is being appealed to here. I feel this section tells us little about what actually useful learners would do in this situation.

  Practically, we’ve converged on pretraining plus episodic reinforcement learning with a relatively secure reward channel to shape the behaviour of our systems. There are reasons to think this is a relatively effective, though not perfect, solution to the reward-as-signalling problem: episodic reward makes reward judgement relatively easy, and pretraining on a shared signal may make generalisation relatively easy to predict. Thus the world currently looks like one where the signalling requirement plays quite an active role in shaping the kinds of systems we develop. Now you can reasonably contend that episodic + predictable generalization is not the endgame, but you need to acknowledge it is sticky because it solves the signalling problem, and any broadening of the scope is only plausible if it continues to solve the signalling problem.

  I think AIT and Bayesian learning and acting plausibly remains a great toolkit to attack the problem, but the problem is fundamentally shaped like an assistance game, not a history based reinforcement learning game. I would be quite interested in seeing a sequence of agents which execute a “sharp left turn”—i.e. for n ⇐ N each agent solves a more difficult assistance game/​solves an assistance game better, while agent N exhibits AIXI-like signal insensitivity (especially if this sequence looks like what we actually see, i.e. it at least begins with episodic RL with broadening horizons and relatively predictable generalization). But this still requires that we make solving the assistance game a core requirement of our agents under analysis.

• David Johnston 16 Jul 2026 21:34 UTC

  1 point

  0

  in reply to: Thao Pham’s comment on: Can we build an early warn­ing sys­tem for loss of con­trol to AI?

  Regarding question 1: “Uncooperativeness” is sort-of a free parameter here, at least in the way the model functions. If you can build an evaluation whose quality scales sufficiently quickly with AI capability, you can use that as your effective measure of uncooperativeness. I think for the policy feedback aim, credibility and consensus are also really important. So the practical limitation is more about what kind of evaluation pipelines you can build, and what kind of consensus you can build around them.

  I am personally sympathetic to more ambitious “alignment” targets, but I think consensus is may be a limiting factor on how ambitious you can get.

  That said, it’s not entirely a free parameter; if you don’t take a stand on traits that are effectively self propagating then you may add a bunch of uncertainty to the self propagation term, which makes the model worse. Similarly if you know a trait undermines your monitoring efforts then not calling it “uncooperative” basically invalidates the broader project.

  To directly answer your question: yes you distinguish “good cooperativeness” from “bad cooperativeness”, “cooperativeness” here is specifically “cooperativeness with the intent to maintain oversight”, not “cooperativeness with anything at all”.

  Regarding question 2: Yes, I’m assuming the detector is not becoming less precise (or becoming more precise); I breezily swept that under “regularity conditions”. But when I think about answering your question, I wonder if it wouldn’t be better to get a sequence of judgement diffs between older and newer detectors, and then you could track the “true positive according to SOTA” and “false positive according to SOTA” trends separately.

Can we build an early warn­ing sys­tem for loss of con­trol to AI?

• David Johnston 16 Jul 2026 0:42 UTC

  7 points

  5

  in reply to: LGS’s comment on: Our re­sponse to Séb Krier on Plan A

  I thought both this piece and Séb’s were unreasonable in multiple places. E.g. I thought this was tendentious:

  As for our plan baking in too much and not leaving room for trial-and-error/​learning by doing, it’s exactly the opposite. Plan A buys us ten more years than we otherwise would’ve had to experiment on our AIs, understand how they work, and try out different regulatory approaches before we proceed to superintelligence.

  Plan A does in fact bake in some pretty strong policy commitments. “Preventing progress is the trial-and-error friendly policy regime” would be an absurd position to take about policy stances toward technological progress in general. The point is not entirely without merit—AI development could outrun policy cycles (and arguably already is) - but the framing strikes me as pretending not to appreciate Séb’s position, and that in turn make me less trusting toward the rest of the piece.

• David Johnston 14 Jul 2026 23:51 UTC

  1 point

  0

  in reply to: StanislavKrym’s comment on: Our re­sponse to Séb Krier on Plan A

  I did say “if I were trying to do this today”. We might disagree about how relevant this is: I think “what I would prefer today” has a little more evidential weight than “plausible exploration of what might happen in the future” (though neither have all that much weight), perhaps you think it doesn’t.