Tom Davidson

Plan A’s prob­lem with dry tinder

• Tom Davidson 9 Jul 2026 12:09 UTC

  3 points

  0

  in reply to: Tom Davidson’s comment on: You can’t imi­ta­tion-learn how to con­tinual-learn

  One Q I have:

  How would this change if we augmented the Transformer so the first layer can “look back” at the final layer from the previous token?

  Optimist: Now the first layer activations can express very complicated concepts that involve many steps of sequential sequencing. So perhaps this helps a lot.

  Sceptical: but from the perspective of the weights that produce the first layer activations, all the new concepts still look like gobbledegook.

  Optimist: True, but in this set-up the the weights are describing the learning rule, they’re not trying to “understand” the object-level content at all. It is the activations that come to “understand” the new concepts.

  Well, if it’s doable after this one simple architectural change, why not via chains of thought like Alex was saying?

  • Firstly, you can send back detailed and novel representations from the final layer activations, but can’t do either from a new token.

  • Second, consider the specific weights that create the first layer activations.

    • With the architectural change, there are specialised weights trained to give sophisticated novel representations to the first layer (stealing them from the final layer of the previous token).

    • But with current transformers, the same weights that map words to existing concepts (that map the word “dog” to a complex representation of dogs in concept space) would have to do double duty and also help create sophisticated novel concepts.

  I think these two differences are pretty big.

• Tom Davidson 9 Jul 2026 11:36 UTC

  7 points

  2

  in reply to: Alex Mallen’s comment on: You can’t imi­ta­tion-learn how to con­tinual-learn

  Flagging that I think this back-and-forth is very important.

  The “limited serial depth” point seems like the core of Steven’s argument for “can’t imitation learn how to continual-learn”. (If he thinks there’s other important parts, i’d be interested to hear it!) And so the more natural statement of the conclusion to me would be “transformers can’t do continual learning at runtime, even if they’ve been trained on trajectories of continual learning”

  Fwiw, i feel pretty convinced by Steven’s argument here. Human continual learning in fact involves parameter changes with massive serial depth. It seems likely that an alg that tries to compress that into 10 serial steps of new concept formation will be extremely inefficient and fail in practice.

• Tom Davidson 9 Jul 2026 10:36 UTC

  LW: 3 AF: 1

  0

  AF

  in reply to: Steven Byrnes’s comment on: Why we should ex­pect ruth­less so­ciopath ASI

  The first one (bootstrapping) has the issue that if the serial thinking is not 100% perfect, then it will sometimes get mistakes, and then you’re SFT’ing on the mistakes, making the model more confident in those mistakes, and then the next round of serial thinking will incorporate and build on those mistakes. Repeat a billion times in a sealed box, and I think it would spiral into nonsense—it would get dumber not smarter.

  Thanks, this is helpful and not an argument i’ve come across before!

  One quick clarification: I presume you’re here talking about systematic mistakes. We can probably both agree that if there are sometimes random mistakes but they are not systematic, then this would be fine.

  I agree that there will be some systematic mistakes if you just think for ages, and that you will then be training that into the next model. But the next model will also have certain advantages. It could be smarter, have better epistemics, and be better at questioning its own biases. (Assuming you try to use serial thought to create data with these qualities!) Those advantages might allow it to recognise the systematic mistake that it has previously been making.

  One way of thinking about this is that there could be some basin of reasonableness and cleverness that the model falls into, where it’s able to recognise and counteract its own systematic biases. And from there it can continually increase its cleverness and reasonableness futher

  I agree that the bureaucracy/​scaffold approach can’t go all the way alone, and would just be an amplifier on this serial thinking time bootstrapping approach.

  And the third approach I referenced was just combos of serial thinking bootstrapping, scaffolds, and bits of RL (but where RL isn’t dominate).

• Tom Davidson 9 Jul 2026 9:55 UTC

  2 points

  0

  in reply to: Steven Byrnes’s comment on: The na­ture of LLM al­gorith­mic progress

  I guess i’m saying that doing that experiments at small scale could very easily cause large inaccuracies, and this is a plausible reason you might be underestimating progress in your 1.1 bucket!

• Tom Davidson 29 May 2026 17:07 UTC

  1 point

  0

  in reply to: callumzc’s comment on: How can the mid­dle pow­ers avoid get­ting trounced dur­ing the in­tel­li­gence ex­plo­sion? A plan.

  Cool! I think the UK could play a leading role here, convening a middle powers coalition and leading on its strategy

• Tom Davidson 28 May 2026 17:58 UTC

  LW: 12 AF: 9

  −2

  AF

  in reply to: Tom Davidson’s comment on: Full au­toma­tion of AI R&D prob­a­bly yields a large speed up even with­out a soft­ware-only singularity

  I thought about this a bit more.

  tldr: Under a simple model and reasonable assumptions, if we automate AI R&D and compute growth stays constant then the pace of AI software progress is 3-5X faster. This means the pace of overall AI progress would be 2-3X faster.

  Assume AI software R&D is Cobb Douglas:

  • g_S = L^alpha E^beta S^-1/​r

  • L = cognitive labour

  • E = experimental compute

  • r governs ideas getting harder to find as you ramp up both cognitive labour and experimental compute. (Note I’ve often confusingly defined r as the returns when you just ramp up cognitive labour—really I should call that r_cog = r * alpha.)

  In this system, before automating away humans, it turns out that:

  • if L grows exponentially at g_L then (holding E fixed) g_S = g_L * alpha * r

  • if E grows exponentially at g_E then (holding L fixed) g_S = g_E * beta * r

  When you automate AI R&D, L = S. In this situation, it turns out that:

  • if L grows exponentially at g_L (eg due to exogenously increasing compute), then:
    g_S = g_L * alpha * r /​ (1 - alpha * r)

    • And if r * alpha > 1 we get a software-only intelligence explosion!

  • if E grows exponentially at g_E (eg due to exogenously increasing compute), S grows at:
    g_S = g_E * beta * r /​ (1 - alpha * r)

  In other words, even absent an SIE, automating AI R&D boosts the standard growth rates by a factor of 1 /​ (1 - alpha * r) due to the fizzling feedback loop of “better software → better AI researchers → better software”.

  This model allows us to ballpark how much faster overall AI progress would be in a regime with full automation but no SIE.

  That regime causes two changes:

  1. g_L gets faster. Firstly, compute growth is somewhat faster than the growth of human AI researchers. Secondly, L is superlinear in compute bc you can run faster and smarter models with more compute.

  2. All growth rates are boosted by a factor of 1 /​ (1 - alpha * r). Ryan used an estimate of r_cog = r * alpha = 0.7. So this is a boost by a factor of 1 /​ (1 − 0.7) = ~3.

  Concretely, let’s make the following assumptions:

  • g_L = g_E today (conservatively assume compute and human researchers have grown at the same pace)

  • alpha = beta = 0.5

  • r = 1.4 (so that we recover Ryan’s r * alpha = 0.7)

  • After automating AI R&D, L grows twice as fast as compute: g_L = 2 * g_E

  Then today the total pace of software progress is :

  g_S = progress due to growing labour + progress due to growing compute

  = g_E * alpha * r + g_E * beta * r

  = g_E * r

  And after AI R&D automation g_L is 2X faster and everything gets boosted by a factor of 3. The pace of software progress is:

  g_S = progress due to growing labour + progress due to growing compute

  = (2 * g_E * alpha * r + g_E * beta * r)*3

  = 4.5 * g_E * r

  So that’s 4.5x faster software progress, holding the rate of compute growth constant! If software and compute contribute equally to AI progress, that’s a bit under 3x faster total AI progress.

  I think not too surprising, given we’re using an r*alpha value close to 1.

  If we used r*alpha =0.5, our boost factor shrinks to 2 and we’d get 3x faster software progress and 2x faster total AI progress.

• Tom Davidson 28 May 2026 13:45 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: ryan_greenblatt’s comment on: Full au­toma­tion of AI R&D prob­a­bly yields a large speed up even with­out a soft­ware-only singularity

  This isn’t what I want because

  yeah i adjust for this in my other comment

  I don’t understand, the whole point of the experiments is to get us better labor.

  yeah, so they do—a doubling of cumulative experiments drives 0.7 doublings of software. And then that better software does more cognitive work to improve software further still. But it doesn’t increase the amount of compute available for experiments, so the feedback loop doesn’t go full circle.

  For “cognitive labour” we have: more compute → more cog labour → beter software → more cog labour → better software… So you get and initial boost from extra compute which then ratchets up with the software feedback loop

  But for “experimental compute” we have: more compute → more experiments → better software → more cog labour → better software...

  So while it feeds into the software feedback loop, it doesn’t loop back to “more experimental compute”.

  I’m not sure if my math full priced this in. I’ll think about that more.

• Tom Davidson 28 May 2026 13:40 UTC

  2 points

  0

  in reply to: Oliver Sourbut’s comment on: Tom David­son’s Shortform

  Thanks for the push—published!

How can the mid­dle pow­ers avoid get­ting trounced dur­ing the in­tel­li­gence ex­plo­sion? A plan.

• Tom Davidson 27 May 2026 21:52 UTC

  LW: 2 AF: 1

  0

  AF

  on: Full au­toma­tion of AI R&D prob­a­bly yields a large speed up even with­out a soft­ware-only singularity

  while 4x more compute at current margins is pretty close to as good as getting compute that’s 4x serially faster

  Why?

• Tom Davidson 27 May 2026 21:49 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: Tom Davidson’s comment on: Full au­toma­tion of AI R&D prob­a­bly yields a large speed up even with­out a soft­ware-only singularity

  In fact the above is conservative in assuming one compute doubling yields just one labor doubling. You get that by running more copies. But you’ll train smarter models

  If one compute doubling yields two labor doublings (holding software constant) then, rerunning the above analysis:

  • One compute doubling → two labor doublings (constant software) → in equilibrium, 6 labor doublings drive 4 software doublings (r=0.7)

  • today one compute doubling is associated with one (or fewer) labor doublings. In the new equ it’s associated with 6

  How much faster would ai progress be compared to today?

  • Suppose that today labor and compute due exp are equally important and produce progress via cobb Douglas. Suppose they grow at equal rates

  • Then with full automation and r=0.7, labor is growing 6x faster than before. And overall ai progress is 3.5x faster

  Of course, compute may be growing more slowly

• Tom Davidson 27 May 2026 21:05 UTC

  LW: 5 AF: 2

  0

  AF

  on: Full au­toma­tion of AI R&D prob­a­bly yields a large speed up even with­out a soft­ware-only singularity

  I haven’t yet found a nice and clean way to model this in isolation

  Not sure if this is what you want, but:

  • Assume r= 0.7

  • Assume that all that doubling compute does is double labor

  • --> in equilibrium, doubling compute doubles software 2 times

  • Why? bc with r=0.7 you need to double labor inputs 3 times in order get 2 software doublings. Which matches the equ: we got 3 labor doublings (one from compute and two from software); and this led to 2 software doublings

  • One way to think of this: today labor grows more slowly than compute. But I’m this world it grows 3x faster.

  • (I can explain better if this is unclear)

  • The above is conservative: it assumes the doubling compute doubles ONLY labor, rather than also doubling experiments

  • If it also doubles experiments, and experiments and labor are equally important, then I believe that the extra doubling of experiments gives us an extra 0.7 doublings of software. (It doesn’t compound further bc there’s no feedback loop to more experiments like there is to more labor)

  • So overall, a compute doubling yields 2+0.7= 2.7 software doublings.

• Tom Davidson 27 May 2026 20:42 UTC

  LW: 2 AF: 1

  0

  AF

  on: Full au­toma­tion of AI R&D prob­a­bly yields a large speed up even with­out a soft­ware-only singularity

  There is a gradual boost setting that smooths out the automation returns over a longer period, but I think this period is unrealistically long such that you don’t see one-time speed-up effects

  Would it help if we added another param controlling how many years the boost occurs over?

• Tom Davidson 21 May 2026 16:23 UTC

  60 points

  17

  on: Tom David­son’s Shortform

  How can the middle powers avoid getting trounced during the intelligence explosion? A plan.

  Superintelligence will likely be developed by US companies; run on US datacentres; and be under the jurisdiction of the US government. This will massively boost US’ military power and make the US economically dominant (eg US producing 99% of world GDP). By default, middle powers will be left in the dust.

  How can middle powers avoid this fate? It’s tough, but here’s the best plan I could think of. (I’m particularly thinking about liberal democracies with influence over AI like UK, Europe, Japan, South Korea, Taiwan.)

  On a very high-level: middle powers should leverage the fact that the US needs them to beat China. It’s genuinely unclear which country will develop superintelligence first, and which would win in a subsequent industrial explosion. Middle powers should help the US, and make sure they are rewarded with continued access to frontier AI and new technologies (including military tech).

  That final boded part is hard. What can the UK realistically do if the US denies it access to frontier AI? The middle powers need a credible alternative to being supplicants of the US. The only alternative that makes sense to me is siding with China. If the US won’t grant middle powers access to their frontier AI, but China will, why should middle powers continue to send AI chips to the US? Why should they continue to support the US diplomatically and militarily? They shouldn’t. They should be willing to pivot to China if the US doesn’t offer AI access sufficient for their national security needs.

  My plan for the middle powers has two stages:

  1. Maintain as much economic and military leverage as possible during the intelligence explosion.

  2. Use that leverage to ensure that, when superintelligence is developed, it refuses to help the US (/​China) disempower the middle powers.

  Stage 1 could well be enough by itself. Maybe middle powers can maintain significant economic and military power indefinitely. But if not, stage 2 is a back-up: it binds the US so that it can’t use its dominance to crush the middle powers.

  I’ll walk through each stage in turn.

  Stage 1: Maintain as much economic and military leverage as possible during the intelligence explosion

  The biggest lever here is securing access to frontier AI. Anton Leicht has a great post about how this is under threat, as evidenced by developments with Mythos. Middle powers should insist on equal commercial terms to US companies, and comparable access for their militaries. This is in AI companies’ interests! A bigger market means more customers and higher prices.

  Aside: why access to frontier AI might be sufficient for middle powers to stay economically relevant indefinitely

  The hope here is that:

  1. Most of the economic surplus from AI is not captured by AI companies. To create economic value, AI must be combined with complementary inputs: factories, human physical labour, know-how of human experts, relationships with suppliers, trusted brands, etc. How much of the surplus will be captured by AI companies vs the owners of these complementary inputs? Optimistically: producers of general-purpose technologies often capture only a small fraction of surplus; and multiple frontier AI companies might sell similar products and bid each other down on cost.

  2. Most of the economic surplus from AI occurs outside the US. The majority of these complementary inputs are situated outside the US. So most AI-driven economic value-add should occur outside US borders.

  If (1) and (2) both hold, a significant fraction of AI’s economic surplus will accrue to non-US actors.

  But how can middle powers guarantee frontier AI access? It’s tough, but a few strategies:

  • Build data centres. Partner with frontier AI companies to build secure data centres domestically, in return for guaranteed frontier access. This is a big win-win. AI companies improve their bargaining position with the US government. Recall, the US government threatened to destroy Anthropic when Anthropic insisted that their AI systems wouldn’t be used for legal mass surveillance.

  • Adopt AI. The more middle powers use frontier AI, the more costly it is for AI companies to cut them off.

  • Invest in frontier AI companies. Once they IPO, middle powers could invest billions or trillions into leading AI companies, in return for access guarantees.

  • Support the US internationally. If middle powers throw their diplomatic and military weight behind US foreign policy objectives, it benefits the US to keep them strong.

  • Build a relationship with China. If the US refuses to grant middle powers access to frontier AI, the national security implications are dire. Middle powers need a plan B, and China is the only other game in town for frontier AI. Only if this alternative is truly credible can it be leveraged into access to US frontier AI.

  • • Ultimately, this involves middle powers threatening to sell semiconductor equipment and chips to China instead of the US. Obviously, that’s pretty far outside the Overton window. But that may change as the world rapidly wakes up to powerful AI and its national security implications.

  • Demand kill switches on US data centres. This is much more late-stage, after the world has truly woken up to the strategic implications of AGI. Suppose US and middle powers agree to a “chips for frontier access” deal – middle powers continue to supply the US with frontier chips; US continues to give middle powers access to frontier AI. The middle powers might still worry: what if the US suddenly changes its mind once it has superintelligence? By then, the US might be powerful enough to dominate without continued allied support. This is where kill switches can help. If the US withdraws AI access, allies could destroy US data centres in response. It’s a way to lock-in the deal.

  • • (h/​t AI futures project for this idea. A related idea is for US data centres to be placed in a location that’s easy to attack – like in space)

  Beyond securing access to frontier AI, how else can middle powers maintain economic and military leverage?

  • Build physical infrastructure. Factories, robots, solar panels, batteries, semiconductors — all these industries are highly complementary to powerful AI.

  • Maintain nuclear 2nd strike capability. The point isn’t to use it. But it improves their leverage for stage 2.

  The catch-all meta-point here is waking middle powers up to superintelligence.

  I’m not recommending middle powers do their own frontier AI development. Seems very hard for them to catch up with the US.

  Stage 2: Ensure that, when superintelligence is developed, it refuses to crush middle powers

  If stage 1 goes well, middle powers remain somewhat powerful economically and militarily deep into the singularity. But it might fail. What can middle powers do if they see the US on track to total global dominance?

  First, they should demand a pause/​slowdown of AI development. But the US may refuse – pausing is very costly if alignment risk is low. And pausing is a stopgap: eventually, superintelligence will be developed.

  An additional demand: when superintelligence is developed, it’s designed to refuse to crush middle powers. By doing this, the US would credibly bind itself to maintaining the sovereignty of other nations.

  The optimistic case here is that this isn’t a big sacrifice for the US. They can still become as rich as they like and achieve their security interests. Sure, they can’t seize control of other nations, but that is not an important goal of theirs anyway. Losing that option is well worth the benefits: other nations cooperate economically, don’t attack US data centres, and don’t threaten nuclear war.

  The pessimistic case is that this involves an insane degree of irrevocable hand-off to AI. The US must literally be unable to attack middle powers no matter how hard it tries: retraining the AI, turning it off, training a new more powerful AI, passing new laws, using the military to destroy the datacentres the AI is running on. For it to be truly binding, the US must permanently hand over military and political power to AI. That might be deeply unpopular, and indeed seem insane to the US. It’s also very hard to verify: you can’t just verify the training run, you need to verify that humans+other AIs have no way to disempower the trained AI. It’s more like verifying “who would win this civil war” than “technical property XYZ holds”.

  The realistic path here probably involves gradually handing off more and more control to AI that refuses to crush middle powers, with no clear point at which humans could no longer wrest back control.

  The longer middle powers wait to push for stage 2, the less leverage they will have because the US will have pulled further ahead economically and militarily. So they should be pushing in this direction constantly. Eg demanding transparency into the model specs of powerful AIs deployed in the US government, and arguing that powerful military AI should be designed to obey international law.

  (I described the plan as involving two stages because that’s how I expect it to play out over time. But succeeding at either stage is sufficient! If middle powers stay economically/​militarily competitive, they never need to bind US superintelligence. And if they do bind superintelligence, they won’t be crushed no matter how far behind they fall.

  Is it good to avoid middle powers getting trounced?

  I live in the UK, so I am biased here. I do not want the UK to become a supplicant to the US!

  But here’s a brainstorm of pros and cons from a more impartial perspective.

  Pros to empowering middle powers:

  • Avoid a single point of failure. If the US becomes globally dominant and its political system fails, that’s a global failure.

  • More democracies. Many middle power democracies look more robust than the US, so more middle powers may mean more democracy.

  • Improve the US. Middle powers will have an interest in maintaining free market democracy in the US. “Free market” bc they’ll want multiple AI companies competing to sell cheap API access to non-US countries. “Democracy” because they’ll expect that the US is more likely to maintain a strong alliance with middle power democracies if it stays democratic.

  • Experimentation. Experimenting with multiple different political and legal systems seems generally good for figuring out a good way to govern society post AGI.

  • Pause AI. They could potentially pressure US/​China to pause/​slowdown reckless AI development.

  • Prosocial norms. When multiple actors bargain with each other (e.g. about how to distribute space resources, whether to develop a dangerous technology), they tend to frame arguments in terms of pro-social norms, and so agreements tend to emphasise the actor’s more virtuous/​ethical values.

  Cons of empowering middle powers. Multipolarity has its own downsides:

  • more likely to lead to war.

  • can drive extreme competition, eg racing to develop a dangerous technology, or to hand off power to misaligned AI

  • Harder to prevent harms from offense-dominant technologies like bioweapons

  • my plan involves waking up middle powers, which could shorten timelines

  What links here?

  • How can the mid­dle pow­ers avoid get­ting trounced dur­ing the in­tel­li­gence ex­plo­sion? A plan. by Tom Davidson (28 May 2026 13:39 UTC; 39 points)

• Tom Davidson 18 May 2026 21:54 UTC

  LW: 2 AF: 1

  0

  AF

  on: Em­pow­er­ment, cor­rigi­bil­ity, etc. are sim­ple ab­strac­tions (of a messed-up on­tol­ogy)

  But alas, I currently think the most important use-case for AGI is figuring out true important things about the world (esp. related to ASI alignment and strategy) and explaining those things to the human. For this process to be effective, we cannot have an AGI that’s unconcerned with what we wind up believing after the discussion—that’s a recipe for slop-and-doom, or just an AI that’s incomprehensible and unhelpful. Rather, I want the AI to be like a disagreeable nerd that wants us to have a good understanding, notices areas where we’re confused, and is brainstorming and strategizing on how to help set us straight by improving its clarity and pedagogy. This strategizing is clearly a form of optimization, and the target of the optimization is related to the human’s eventual desires (well, it’s nominally about the human’s beliefs, but beliefs and desires are entangled)

  What about if AI optimizes for humans fully understanding the evidence and arguments that it has discovered but does not optimize for what humans actually end up believing about the bottom line?

  So if it discovered powerful arguments for misalignment, it would make sure to explain them in a way that the humans fully understand those arguments. But if humans had some prior/​bias that caused them to dismiss the argument even when they fully understand it, the AI would not try and optimize around that.

• Tom Davidson 17 May 2026 12:44 UTC

  1 point

  0

  on: Ir­re­triev­abil­ity; or, Mur­phy’s Curse of Oneshot­ness upon ASI

  If failure of alignment → schemer that wants to seize power, then ASI alignment is one shot.

  But if failure of alignment → non-schemer misalignment (eg reward hacking, or flailing misgeneralisation), then we failure isn’t existential

  So I think p(scheming | alignment failure) is a crux here

A Re­search Agenda for Se­cret Loyalties

• Tom Davidson 7 May 2026 22:38 UTC

  2 points

  0

  on: The na­ture of LLM al­gorith­mic progress

  Fwiw it’s unclear to me whether epoch’s methodology would make this mistake

  After 2018 they have many data points at different times and compute scales. In principle, that should allow them to disentangle alg progress from the effect of compute. If models trained with the the same compute but later in time are no better, they shouldn’t find any alg progress. (But I haven’t thought about this in a while, and the strong correlation between time and compute in their data makes their results super noisy)

• Tom Davidson 7 May 2026 22:04 UTC

  2 points

  0

  on: The na­ture of LLM al­gorith­mic progress

  Here’s my current impression (any uncited claims probably come from @Hans Gundlach et al. 2025b “On the origin of algorithmic progress in AI

  Aren’t their experiments at too small a scale to pick up the benefits?