Tom Davidson

Karma: 1,490

Tom Davidson 7 Dec 2025 13:15 UTC
LW: 4 AF: 2
0
AF
in reply to: Lukas Finnveden’s comment on: The Industrial Explosion
Thanks! (Quickly written reply!)
I believe I was here thinking about how society has, at least in the past few hundred years, spent a minority of GDP on obtaining new raw materials. Which suggests that access to such materials wasn’t a significant bottleneck on expansion.
So it’s a stronger claim that “hard cap”. I think a hard cap would, theoretically, result in all GDP being used to unblock the bottleneck, as there’s no other way to increase GDP. I think you could quantify the strength of the bottleneck as the marginal elasticity of GDP to additional raw materials. In a task-based model, i think the % of GDP spent on each task is proportional to this elasticity?
Though this seems kind of like a fully general argument
Yeah, I think maybe it is? I do feel like, given the very long history of sustained growth, it’s on the sceptic to explain why their proposed bottleneck will kick in with explosive growth but not before. So you could state my argument as: raw materials never bottlenecked growth before; no particular reason they would just bc growth is faster bc that faster growth is driven by having more labour+capital which can be used for gathering more resources; so we shouldn’t expect raw materials to bottleneck growth in the future.
TBC, this is all compatible with “if we had way more raw materials then this would boost output”. E.g. in Cobb Douglas doubling an input to output increases output notably, but there still aren’t bottlenecks.
(And i actually agree that it’s more like CES with rho<0, i.e. raw materials is a stronger bottleneck, but just think we’ll be able to spend output to get more raw materials.)
(Also, to clarify: this is all about the feasibility of explosive growth. I’m not claiming it would be good to do any of this!)

Tom Davidson 17 Nov 2025 18:26 UTC
3 points
0
in reply to: reallyeli’s comment on: How quick and big would a software intelligence explosion be?
Yeah, i think one of the biggest weaknesses of this model, and honestly of most thinking on the intelligence explosion, is not carefully thinking through the data.
During an SIE, AIs will need to generate data themselves, by doing the things that human researchers currently do to generate data. That includes finding new untapped data sources, creating virtual envs, creating SFT data themselves by doing tasks with scaffolds, etc.
OTOH it seems unlikely they’ll have anything as easy as the internet to work with. OTOH, internet data is actually v poorly targeted at teaching AIs how to do crucial real-world tasks, so perhaps with abundant cognitive labour you can do much better and make curriculla that directly targeted the skills that most need improving

Tom Davidson 17 Nov 2025 18:21 UTC
2 points
0
in reply to: reallyeli’s comment on: How quick and big would a software intelligence explosion be?
Yep, the ‘gradual boost’ section is the one for this. Also my historical work on the compute-centric model (see link in post) models gradual automation in detail.
So if you’ve fully ignored the fact that pre-ASARA systems have sped things up, then accounting for that will make takeoff less fast bc by the time ASARA comes around you’ll have already plucked much of the low-hanging fruit of software progress.
But I didn’t fully ignore that, even outside of the gradual boost section. I somewhat adjusted my estimate of r and of “distance to effective limits” to account for intermediate software progress. Then, in the gradual boost section, i got rid of these adjustments as they weren’t needed. Turned out that takeoff was then faster. My interpretation (as i say in the gradual boost section): dropping those adjustments had a bigger effect than changing the modelling.
To put it anothr way: if you run the gradual boost section but literally leave all the parameters unchanged, you’ll get a slower takeoff.

Tom Davidson’s Shortform

Tom Davidson16 Oct 2025 18:11 UTC

6 points

1 comment1 min readLW link

Tom Davidson 16 Oct 2025 18:11 UTC
12 points
0
on: Tom Davidson’s Shortform
Forethought is hiring!
You can see our research here.
You can read about what it’s like to work with us here.
We’re currently hiring researchers, and I’d love LW readers to apply.
If you like writing and reading LessWrong, I think you might also enjoy working at Forethought.
I joined Forethought a year ago, and it’s been pretty transformative for my research. I get lots of feedback on my research and great collaboration opportunities.
The median views of our staff are often different from the median views of LW. E.g. we probably have a lower probability on AI takeover (though I’m still >10% on that). That’s part of the reason i’m excited for LW readers to apply. I think a great way to make intellectual progress is via debate. So we want to hire ppl who strongly disagree with us, and have their own perspectives on what’s going on in AI.
We’ve also got a referral bounty of £10,000 for counterfactual recommendations for successful Senior Research Fellow hires, and £5,000 for Research Fellows.
The deadline for applications is Sunday 2nd November. Happy to answer questions!

Tom Davidson 16 Oct 2025 15:15 UTC
5 points
0
on: A personal take on why you should work at Forethought (maybe)
I also work at Forethought!

I agree with a lot of this post, but wanted to flag that I would be very excited for ppl doing blue skies research to apply and want Forethought to be a place that’s good for that. We want to work on high impact research and understand that sometimes mean doing things where it’s unclear up front if it will bear fruit.

Tom Davidson 8 Oct 2025 9:26 UTC
2 points
0
in reply to: L Rudolf L’s comment on: Sense-making about extreme power concentration
(Fyi the previous comment from “Tom” was not actually from me. I think it was Rose. But this one is from me!)
Worth noting that the “classic” AI risk also relies on human labour not being needed anymore. For AI to seize power, it must be able to do so without human help (hence human labour not needed), and for it to kill everyone human labour must not be needed to make new chips / robots

Tom Davidson 3 Oct 2025 14:12 UTC
9 points
0
in reply to: L Rudolf L’s comment on: Sense-making about extreme power concentration
Thanks, I like this!

Haven’t fully wrapped my head around it yet, but will think more.

One quick minor reaction is that I don’t think you need IC stuff for coups. To give a not very plausible but clear example: a company has a giant intelligence explosion and then can make its own nanobots to take over the world. Doesn’t require broad automation, incentives for governments to serve their people to change, etc

Tom Davidson 3 Sep 2025 0:58 UTC
4 points
0
in reply to: Daniel Kokotajlo’s comment on: Could one country outgrow the rest of the world?
Do you think the same for a company within the US? That with a 6 month, or even just a 3 month going off recent trends, lead it would find a way to sabotage other companies?
I think it’s plausible, but:
- cyber attacks might be detectable and they’re unequivocally illegal
- political manoeuvrings take time and there’s a pretty strong status quo bias in politics (hard to pass legislation)
  - and the govt typically doesn’t private companies to have monopolies on critical tech, and even with a private-public partnership my understanding is govt would want multiple companies involved
  - and the govt might get very involved if there’s a public-private partnership, which could threaten company leadership
(I think a remerging crux here might be the power of AI’s persuasion+strategy skills)

Tom Davidson 3 Sep 2025 0:51 UTC
2 points
−2
in reply to: Daniel Kokotajlo’s comment on: Could one country outgrow the rest of the world?
Thanks for this!
Yep i agree that scenario gets >5%!
Agree ASML is one of the trickiest cases for OpenBrain. Though I imagine there are many parts of the semiconductor supply chain with TSMC-level monopolies (which is less than ASML). And i don’t think hacking will work. These companies already protect themselves from this, knowing that it’s a threat today. Data stored on local physical machines that aren’t internet connected, and in ppl’s heads.
And i think you’ll take many months of delay if you go to ASML’s suppliers rather than ASML. ASLM built their factories using many years worth of output from suppliers. Even with massive efficiency gains over ASML (despite not knowing their trade secrets) it will take you months to replicate.
I agree that the more OpenBrain have super-strategy and super-persuasion stuff, the more likely they can capture all the gains from trade. (And military pressure can help here too, like with colonialism.)
Also, if OpenBrain can prevent anyone else developing ASI for years, e.g. by sabotage, i think they have a massive advantage. Then ASML loses their option of just waiting a few months and trading with someone else. I think this is your strongest argument tbh.
Biggest cruxes imo:
- Size of SIE
- Speed of industrial explosion when the SIE is finishing
  - Depends on size of SIE and on how hard it is to build fast-replicating robots
- How long OpenBrain has a monopoly on ASI (maybe indefinitely via sabotage)
  - Depends on size of SIE, gap with laggard, whether they dare to sabotage
- Whether OpenBrain can take ~all gains from trade with other complementary companies
  - Depends on AI persuasion/strategy after the SIE, whether OpenBrain can quickly rediscover all their insights with way fewer experiments, and whether OpenBrain has a sustained monopoly on ASI
I’m curious how confident you are a company with a 6 month lead could outgrow the rest of the world by themselves?

Tom Davidson 2 Sep 2025 23:42 UTC
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Could one country outgrow the rest of the world?
as discussed on the other thread, i’m imagining other countries have access to powerful AI as well. Or even equally good AI if they are sold access or if another actor catches up after the SIE fizzles (as it must)

Tom Davidson 2 Sep 2025 8:26 UTC
4 points
0
in reply to: Daniel Kokotajlo’s comment on: Could one country outgrow the rest of the world?
Great—I agree that if you can get to >50% physical capital (as valued by how useful it is for the new supercharged economy) within 6 months of the SIE fizzling out, then you can outgrow the world in the scenario you describe. Sounds like you’re more bullish on a very fast industrial explosion than I am—I think it’s hard to know whether we’ll get physical capital doubling times of <3 months within a few months of SIE. On longer timelines to an SIE, this seems more plausible as there’s more time for robotics to improve in the meantime.
Another blocker—cooperation from pre-existing companies
I want to mention another blocker to someone outgrowing the world via a very fast industrial explosion. You’ll be able to go faster if you draw heavily on the knowledge, skills and machines of existing companies. So you’ll need their cooperation. But they might not cooperate unless you give them a big fraction of the economic value. Which would prevent you from outgrowing the world.
Of course, you could “go it alone” and leapfrog the companies that won’t do business with you. But then it’s even harder to do a very fast industrial explosion. E.g. take ASML and TSMC. I predict it will be much more efficient for the leader in AI to work with those companies (e.g. selling them API access to ASI, or acquiring them) than to leapfrog. If they try to leapfrog, the laggard can team up with them and recover some of the lost lead.
Concretely, let’s say OpenBrain gets to AGI first. Then its AIs are combined with ASML’s existing knowledge to create EUV++ machines. Are the profits from these machines going to ASML or to OpenBrain? Seems unclear to me. ASML’s bargaining position might be strong—they have a strong monopoly and can wait 6 months and deal with the laggard instead.
So maybe other companies have a lot of economic leverage post-AGI, and they convert that to economic value. And, as discussed, maybe other governments tax the activites happening in their domains.
Will $ be helpful?
But you suggest these “mere $” might not be very useful, if ASI has colonised space and doesn’t care. But $ can be used to buy AI and buy crazy new physical tech. So if other actors have lots of $, they can convert that to ability to colonise the stars themselves.
Q about your Israel scenario?
Are you basically assuming a merge between the lab and govt here? And then an nation-wide effort to utilise Israel’s existing human and physical capital to kick-start the industrial explosion within it own borders?
Asking bc absent a merge i’m still not seeing a reason for all actors outside Israel to be left in the dust. The AI company would trade with some Israeli companies and (many more) non-Israeli ones. In which case Israel per se would only outgrow the world if the AI company outgrows the world by itself.

Tom Davidson 29 Aug 2025 14:46 UTC
4 points
0
in reply to: Daniel Kokotajlo’s comment on: Could one country outgrow the rest of the world?
Yeah let’s think.
If there’s a AI-2027 style SIE, then 6 months is a gap of ~6 OOMs of effective compute, which is maybe equivalent to a having 1000X more researchers thinking 4X faster and much smarter. (Maybe 2X the gap as from top expert to median expert.)
That’s a big advantage. Much bigger than US had over China over the last 70 years. So yeah it’s pretty plausible that the reverse engineering would be too slow for Germany to catch up using Chinese AI…
Though once the SIE fizzles out Germany will have access to ~equally good AI. And if the industrial explosion has still got many years to go that could give them time to catch up.
There’s also the question of how Germany is able to use the US AI. For example, suppose two US projects are neck and neck. Or suppose there’s one US project, but multiple downstream ‘product’ companies that can sell fine-tune and sell access to the model. In that case, competition could lead for German companies to be able to access AI for a wide range of tasks. Including ‘R&D’ that really involves copying/adapting US tech developed elsewhere. In this case, Germany doesn’t have to use worse AI.
OTOH, maybe there’s just one US project and it’s single-handedly making tech progress in the US and selling its systems abroad. In this case, it could have strong incentive to prevent German companies using its AI to copy US tech. Bc it developed that tech itself! This is a case where the US is strongly coordinated bc all US economic activity is just one company.
Another scenario. There’s one US AGI project. It sells API access to existing companies in the US, at a very healthy profit, and those companies all make amazing new tech and own the IP. In this case, teh US project still has incentive to sell its AI to Germany companies that will use it to copy the tech developed in US companies. (In fact, there’s no reason the US companies would be ahead of the German companies in developing the tech in this scenario.) Here is seems like USG needs to step in to make sure US companies keep the profits.
Like, here’s one way of putting it. There’s def a chance that one project outgrows the world. Especially if there’s a monopoly on AI within the US with the support of the govt. That project would be internally coordinated and avoid trades that disadvantage it relative to RoW. But suppose one project doesn’t outgrow the world. There’s then a question of why the larger entity that does outgrow the world is the US. Like, why are other US companies in on the action, but no non-US companies. ASML and TSMC will have a lot to offer! My answer to this question is that USG would have to make specific efforts, significantly restricting trade, to make this happen.
Then the scenario where amazing tech is built in Germany by US companies. Let’s put aside manipulation and military exploitation for a second. The GDP produced by these companies is German GDP and the German government can tax it. So, if most physical manufacturing is not on US soil and most GDP is from physical stuff being produced, then >25% GDP will be produced on non-US soil and could be taxed by non-US governments. But yeah, i agree that the analogy of colonization is chilling here.
(sorry, this isnt’ very well organised. But overall my tentative position is that it is plausible US outgrows the world here, but it seems less certain and more complicated than what appears to be your position)

Tom Davidson 28 Aug 2025 18:47 UTC
6 points
0
on: Could one country outgrow the rest of the world?
Posting a brief exchange between myself and Sam Winter-Levy on this topic, relating to this piece of his.
Sam
Hi Tom,

Sam Winter-Levy here—I’m a fellow at the Carnegie Endowment for International Peace in DC. Really interesting piece on economic growth explosions. It overlaps with some work we’re doing at Carnegie on “decisive strategic advantages,” just wanted to share some quick reactions in case helpful!

I know you purposefully set aside military considerations, but I’m not sure I agree with the premise that outgrowing the world is a path to a “decisive strategic advantage without acting aggressively towards other countries.” In particular, I think that so long as nuclear deterrence and MAD hold, they are a binding obstacle to any alleged DSA, no matter how much AI turbocharges economic growth. Today, the US economy is 15x Russia’s and 1000x North Korea’s, but the US has no ability to impose “complete domination” over either of them, to put it mildly, nor would it even if AI dramatically boosted the US economy still further—in large part because of Russia and North Korea’s ability to credibly threaten to impose massive costs on the US through nuclear escalation on any issue of sufficient importance to either regime.

More broadly, I don’t think most claims about the existence of a DSA take nuclear deterrence seriously enough. Before nuclear weapons existed, a state’s ability to defend itself from coercion depended crucially on its economic size, since relative forces mattered and force size/quality depended ultimately on having a large population and an advanced economy. But that’s not true once you have a second-strike nuclear capability; nuclear weapons break the connection you emphasize between economic power and the ability to impose your political preferences on other nuclear-armed states. So I think even granting the argument that AGI could lead to some sort of economic explosion for one state, the amount of coercive leverage that would grant that state is likely to be highly limited, nowhere close to the Bostrom DSA definition you use, so long as nuclear deterrence remains intact.

In case of interest, here’s a piece we published recently on these issues, coauthored with Nikita Lalwani (former director for tech/nat sec on the Biden NSC). We make a version of the above argument about the importance of MAD for the existence of any alleged DSA, and then lay out how exactly AGI might threaten nuclear deterrence. Would welcome any feedback! We’re still tentatively thinking through these DSA issues ourselves.

Best,

Sam
Tom
Hi Sam,
Thanks for this great pushback!
I think you’re right that this is an oversight of the piece. But here are some responses:
1. In terms of controlling civilisation’s longrun future, outgrowing the world in the way we describe would be enough. We’ll eventually settle space and North Korea/Russia will be the tiniest fraction of total resources/‘land throughout space’. So if US controls everything but some countries on earth, they completely dominate the longrun future. Agree though the DSA definition i used is a bit different to that
2. If a country gains this kind of massive economic advantage, it will gain a massive tech advantage as well that, at some point, is likely to allow it to remove the second strike capability. This is more speculative, but we are here talking about the limits of technology and 1-month doubling nanobots etc. Still, this at least should be addressed by the post.

Tom Davidson 28 Aug 2025 18:17 UTC
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Could one country outgrow the rest of the world?
Ah, cool.
I was roughly imagining:
- US gets to ASI first
- China 6 months behind
- Chinese AI used in Europe. US AI also used in Europe. Deep integration with the physical economies due to the massive complementarities.
So then the Q is why Europe can’t quickly use its AI to copy the new physical tech?
And similarly for China, which can use its worse AI (and the US AI as well if the status quo continues).
Also, even absent copying the new tech, the amazing new physical robots will be made on European soil. (But maybe US companies still capture the profits?)

Tom Davidson 28 Aug 2025 17:27 UTC
4 points
0
in reply to: Daniel Kokotajlo’s comment on: Could one country outgrow the rest of the world?
Thanks!
The key insight is that ordinary human leaking and spying and tech diffusion happens at ordinary human pace. If e.g. China is doing an intelligence explosion such that it’s making tech and economic progress at 100x normal speed, the process of ideas gradually diffusing to the US and causing catch-up growth won’t be 100x faster to compensate. Instead e.g. the lag between “the chinese are selling an exciting new product” and “our engineers have dissected it and deduced some interesting ideas about how it was made and replicated these ideas in our own manufacturing processes” will be… approximately the same? And so for practical purposes it might as well not be happening.
I don’t think it will be the same? If the economy is growing 100X as fast then that means:
- physical artefacts transported from place to place 100X faster
- new tech and products adopted throughout the economy 100X faster
Both these things help with diffusion between countries as well.
And if designing new tech is 100X faster, i’m not seeing why we would expect copying tech to be <100X faster. With copying, you start with additional clues—e.g. a physical artefact or using a digital product or public materials (like OAI’s o3 demo) -- and design the tech from there. Why would designing new tech from scratch be sped up by more than designing from clues? I.e. we’ve got super-smart AI doing both.
the rate of tech diffusion via leaks and spying will go way down
Yeah, agree spying will go way down once humans are replaced, and cyber hacks will go down too. Though i would have thought a small minority of tech diffusion comes from spies + hacks? E.g. i’d have thought selling and publishing about o1 was more important than those things, though i could be wrong and spies might be unually important for AI diffusion in particular.
...
I don’t think you’ve responded to my argument that companies in the leading country could make way more short-term revenues by producing and selling their technologies in other countries? (See the argument in the box.) Bc ASI cognitive capabilities are massively complementary with human labour and physical capital inputs.

Tom Davidson 28 Aug 2025 11:52 UTC
9 points
0
in reply to: Lukas Finnveden’s comment on: Could one country outgrow the rest of the world?
Why look at the number immediately after AGI, here, rather than further into the intelligence explosion?
Here was the thought:
If the lab hasn’t outgrown RoW by the time RoW develops AGI then we’re now in a position where RoW has more physical capital than the lab. This means that, in the next physical doubling, RoW will produce more physical capital than the lab, and so do more learning by doing than the lab has ever done. After that doubling, RoW will then have better tech than the lab and so their physical capital will double more quickly. So, after that doubling, RoW will have both more physical capital and their physical capital will double more quickly. This means they’re on a trajectory to always have more physical capital and the lab can’t outgrow them.
So this is implicitly assuming that the number of units produced is the hard bottleneck on the technology of physical capital, and the lab’s additional AI cognitive labour doesn’t help. This is in line with the classic economist view that you bump up against ultimate limits to what can be inferred from data. But I think qualitatively smarter AI could change this a lot. Maybe the lab has much smarter AI than RoW and so can learn much more per unit produced.
So, thinking about it more, I think this calc makes sense if we start the clock at the point when the SIE (software-only intelligence explosion) fizzles out. Because then, once RoW catches up, they’ll have ~equal cognitive inputs to the lab. So the lab really does need to overtake RoW on physical capital before this point. (Maybe the lab can get more chips than RoW? But this is a type of physical capital.)
Anyway, accounting for this makes it more plausible the lab can outgrow RoW. They need the 1 month doubling time robots not when they first have AGI, but when the software intelligence explosion has fizzled. Which means they have more powerful AI and longer to do robot experiments.
Another reason my calc understates the lab’s chance is that there may be higher value from experiments conducted in serial. RoW constructs all their physical capital in parallel in one doubling. Whereas the lab does it over multiple doublings one after the other. That could make a big difference.
How to do better?
The more nuanced way to game this out would be to represent, at each time-step, the physical capital, cognitive labour, and technology (for physical capital) of both the lab and RoW.
Then have a production function for how inputs of physical capital produced and cognitive labour improve technology in each time-step. Probably smg like g_A = (K^a * C^b)^lambda * A^-beta. Where a+b=1, lambda represents the benefits of doing research in serial, and beta accounts for ideas getting harder to find. (This accounts for your other point about the rate of production doubling every 3 doublings. The parameter beta controls whether it’s “3” or some other number.)
Then in each timestep simulate each actor’s change in:
- Physical capital
  - the level of technology determines the physical capital doubling time
  - so you combine current physical capital and technology to get new physical capital
- Technology
  - Apply the equation above for g_A.
- Cognitive labour.
  - Probably simplest to make this exogenous. Specify some trajectory given your views on the software-only intelligence explosion.
My overall view
- I think that physical capital doubling times of a few months shortly after AGI are more plausible than when I first wrote that appendix. So I think this is plausible if we’re starting the clock at the end of the SIE.
- Accounting improving serial experiments will favour the lab further. As will the possibility the lab can maintain a cognitive advantage beyond the SIE by designing and quickly manufacturing amazing chips.
- But the assumption that a company can spend $1trillion on physical capital before the end of the SIE still seems too high. Though it’s unrealistic to think RoW will use all their physical capital for growth, which balances this out.
- The assumption of a 9 month lead feels generous.
Maybe the most plausible scenarios involve ~6 month leads, ~$250b on physical capital, ~1 month robot doubling times at the end of the SIE, and doubling times becoming faster over time.
I made a simple of the dynamic here. (Still not modelling the gains from serial experiments and
So overall I think I’d still bet against the lab being in a position to do this on the economic fundamentals; but it is more plausible that I’d thought.
What links here?
- Could one country outgrow the rest of the world? by Tom Davidson (21 Aug 2025 15:32 UTC; 45 points)

Tom Davidson 28 Aug 2025 10:19 UTC
3 points
0
in reply to: Lorentz Manifold’s comment on: The Industrial Explosion
Thanks for this helpful comment.
Some quick follow-up Qs.
First:
Of course, scale does have an effect; larger animals have slower metabolisms (smaller specific surface area, difficulty with heat dissipation). However, you can see that there is no order-of-magnitude difference in metabolic rate between a hummingbird and an elephant, so this is not the primary contributing factor.
GPT-5 thought that the ratio of metabolic rates was 100X. Two OOMs!
Second, you say that nanotech could only replicate very fast if it harvests the ready-made compounds in the biosphere. But plant cells that make their food from photosynthesis can replicate in hours/days. E.g. duck weed.
Third, I’d be interested if you had sources for your claims against J/kg and W/kg

Tom Davidson 28 Aug 2025 10:04 UTC
2 points
0
in reply to: David Duvenaud’s comment on: Thoughts on Gradual Disempowerment
In the absolute Malthusian limit AI won’t earn more than its running costs and can’t save. So if we expect AI wages to approach that limit, that seems like a strong reason not to expect that humans will keep more money than AI

But yeah, wages probably won’t be completely up against that limit in practice, as for humans throughout history. But i think it might be pretty close

Tom Davidson 28 Aug 2025 9:54 UTC
2 points
0
in reply to: David Scott Krueger (formerly: capybaralet)’s comment on: Thoughts on Gradual Disempowerment
Thanks!
It feels like this statement should be qualified more… this is holding GD to a higher bar
Yeah fair. I’ve edited to qualify it more.

The analysis of economic disempowerment seems to take place in a vacuum, ignoring one of the main arguments we make, which is that different forms of disempowerment can mutually reinforce each other. The most concerning version of this, I think, is not just “we don’t get UBI”, but rather that the memes that say “it’s good to hand over as much power as quickly as possible to AI” win the day.
Yeah it’s a bit tricky to know how to structure the argument when you’re saying that the 3 domains all mutually reinforce each other. Like, after reading the paper it was unclear to me why the 3 domains don’t mutually reinforce each other to remain good given that they start good. The order in which the paper’s sections appear, and the arguments within, suggested that the main mechanism was:
- Humans lose significant economic power
- Then they start losing cultural power and influence over the state
- Then they have a small amount of each that they ultimately are disempowered in all domains
And so a natural response (which i gave) is:
- Humans will keep economic power
- As a result, they’ll keep cultural influence and control of the state
But yeah, i agree cultural shifts to favour hand off will happen even absent economic disempowerment. I think just driven by ordinary economic competition. And if we hand off to sufficiently misaligned AI, we’re screwed. Assuming AI is aligned enough that it never seeeks power, i’m not sure how worried we should be about handoff. But plausibly we should demand a higher alignment bar than that.
Anyway, to my mind this argument is better understood as “competitive pressure to hand off to misaligned AI” than as an interplay between economic and cultural and state disempowerment, but I do buy it.

The analysis of cultural disempowerment goes one step “worse”, arguing that “If humans remain economically empowered (in the sense of having much more money than AI), I think they will likely remain culturally empowered.” I think we agree that a reasonable model here is one where cultural and economic are tightly coupled, but I don’t see why that means they won’t both go off the rails. You seem to think that they are almost guaranteed to feedback on each other in a way that maintains human power, but I think it can easily go the opposite way.
Yeah, to clarify, i don’t feel it’s guaranteed to maintain human power. Overall, I feel like “yeah i guess maybe that could happen, though none of the mechanisms you mention seem that convincing and there seem like counter considerations and humans will have a strong incentive to keep power if they can and (hopefully!) truthful AI advice to help them and myopically aligned AIs to implement things to help… and also we can see this playing out in real time and respond so not sure it’s worth focussing on in advance (though i agree it will be worth focussing on while it’s happening)”.
Do you think that, absent AI power-seeking, this dynamic is highly likely to lead to human disempowerment? (If so, then i disagree.)
Regarding political disempowerment, you state: “It’s hard to see how those leading the state and the top AI companies could be disempowered, absent misalignment.” Personally, I find this quite easy. Insufficient elite coordination is one mechanism (discussed below). But reality can also just be unfriendly to you and force you to make choices about how you prioritize long-term vs. short-term objectives, leading people to accept deals like: “I’ll be rich and powerful for the next hundred years, and then my AI will take over my domain and do as it pleases”. Furthermore, if more people take such deals, this creates pressure for others to do so as well, since you need to get power in the short-term in order to remain “solvent” in the long term, even if you aren’t myopic yourself. I think this is already happening; the AI arms race is burning the commons every day; I don’t expect it to stop.

I said “absent misalignemnt”, and I think your story involves misalignment? Otherwise the human could hand off to AI that represents their interests. Clearly there’s a problem with handoff if AI seeks power. And i agree it seems bad if AI doesn’t seek power that but also won’t represent human interests as it governs. Though i feel a bit confused about how humans are never able to coordinate to reign it all back in if AIs aren’t seeking power.

Tom Davidson

Tom David­son’s Shortform

Tom Davidson’s Shortform