Tom Davidson

Karma: 1,446

Tom Davidson 3 Sep 2025 0:58 UTC
4 points
0
in reply to: Daniel Kokotajlo’s comment on: Could one country outgrow the rest of the world?
Do you think the same for a company within the US? That with a 6 month, or even just a 3 month going off recent trends, lead it would find a way to sabotage other companies?
I think it’s plausible, but:
- cyber attacks might be detectable and they’re unequivocally illegal
- political manoeuvrings take time and there’s a pretty strong status quo bias in politics (hard to pass legislation)
  - and the govt typically doesn’t private companies to have monopolies on critical tech, and even with a private-public partnership my understanding is govt would want multiple companies involved
  - and the govt might get very involved if there’s a public-private partnership, which could threaten company leadership
(I think a remerging crux here might be the power of AI’s persuasion+strategy skills)

Tom Davidson 3 Sep 2025 0:51 UTC
2 points
−2
in reply to: Daniel Kokotajlo’s comment on: Could one country outgrow the rest of the world?
Thanks for this!
Yep i agree that scenario gets >5%!
Agree ASML is one of the trickiest cases for OpenBrain. Though I imagine there are many parts of the semiconductor supply chain with TSMC-level monopolies (which is less than ASML). And i don’t think hacking will work. These companies already protect themselves from this, knowing that it’s a threat today. Data stored on local physical machines that aren’t internet connected, and in ppl’s heads.
And i think you’ll take many months of delay if you go to ASML’s suppliers rather than ASML. ASLM built their factories using many years worth of output from suppliers. Even with massive efficiency gains over ASML (despite not knowing their trade secrets) it will take you months to replicate.
I agree that the more OpenBrain have super-strategy and super-persuasion stuff, the more likely they can capture all the gains from trade. (And military pressure can help here too, like with colonialism.)
Also, if OpenBrain can prevent anyone else developing ASI for years, e.g. by sabotage, i think they have a massive advantage. Then ASML loses their option of just waiting a few months and trading with someone else. I think this is your strongest argument tbh.
Biggest cruxes imo:
- Size of SIE
- Speed of industrial explosion when the SIE is finishing
  - Depends on size of SIE and on how hard it is to build fast-replicating robots
- How long OpenBrain has a monopoly on ASI (maybe indefinitely via sabotage)
  - Depends on size of SIE, gap with laggard, whether they dare to sabotage
- Whether OpenBrain can take ~all gains from trade with other complementary companies
  - Depends on AI persuasion/strategy after the SIE, whether OpenBrain can quickly rediscover all their insights with way fewer experiments, and whether OpenBrain has a sustained monopoly on ASI
I’m curious how confident you are a company with a 6 month lead could outgrow the rest of the world by themselves?

Tom Davidson 2 Sep 2025 23:42 UTC
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Could one country outgrow the rest of the world?
as discussed on the other thread, i’m imagining other countries have access to powerful AI as well. Or even equally good AI if they are sold access or if another actor catches up after the SIE fizzles (as it must)

Tom Davidson 2 Sep 2025 8:26 UTC
4 points
0
in reply to: Daniel Kokotajlo’s comment on: Could one country outgrow the rest of the world?
Great—I agree that if you can get to >50% physical capital (as valued by how useful it is for the new supercharged economy) within 6 months of the SIE fizzling out, then you can outgrow the world in the scenario you describe. Sounds like you’re more bullish on a very fast industrial explosion than I am—I think it’s hard to know whether we’ll get physical capital doubling times of <3 months within a few months of SIE. On longer timelines to an SIE, this seems more plausible as there’s more time for robotics to improve in the meantime.
Another blocker—cooperation from pre-existing companies
I want to mention another blocker to someone outgrowing the world via a very fast industrial explosion. You’ll be able to go faster if you draw heavily on the knowledge, skills and machines of existing companies. So you’ll need their cooperation. But they might not cooperate unless you give them a big fraction of the economic value. Which would prevent you from outgrowing the world.
Of course, you could “go it alone” and leapfrog the companies that won’t do business with you. But then it’s even harder to do a very fast industrial explosion. E.g. take ASML and TSMC. I predict it will be much more efficient for the leader in AI to work with those companies (e.g. selling them API access to ASI, or acquiring them) than to leapfrog. If they try to leapfrog, the laggard can team up with them and recover some of the lost lead.
Concretely, let’s say OpenBrain gets to AGI first. Then its AIs are combined with ASML’s existing knowledge to create EUV++ machines. Are the profits from these machines going to ASML or to OpenBrain? Seems unclear to me. ASML’s bargaining position might be strong—they have a strong monopoly and can wait 6 months and deal with the laggard instead.
So maybe other companies have a lot of economic leverage post-AGI, and they convert that to economic value. And, as discussed, maybe other governments tax the activites happening in their domains.
Will $ be helpful?
But you suggest these “mere $” might not be very useful, if ASI has colonised space and doesn’t care. But $ can be used to buy AI and buy crazy new physical tech. So if other actors have lots of $, they can convert that to ability to colonise the stars themselves.
Q about your Israel scenario?
Are you basically assuming a merge between the lab and govt here? And then an nation-wide effort to utilise Israel’s existing human and physical capital to kick-start the industrial explosion within it own borders?
Asking bc absent a merge i’m still not seeing a reason for all actors outside Israel to be left in the dust. The AI company would trade with some Israeli companies and (many more) non-Israeli ones. In which case Israel per se would only outgrow the world if the AI company outgrows the world by itself.

Tom Davidson 29 Aug 2025 14:46 UTC
4 points
0
in reply to: Daniel Kokotajlo’s comment on: Could one country outgrow the rest of the world?
Yeah let’s think.
If there’s a AI-2027 style SIE, then 6 months is a gap of ~6 OOMs of effective compute, which is maybe equivalent to a having 1000X more researchers thinking 4X faster and much smarter. (Maybe 2X the gap as from top expert to median expert.)
That’s a big advantage. Much bigger than US had over China over the last 70 years. So yeah it’s pretty plausible that the reverse engineering would be too slow for Germany to catch up using Chinese AI…
Though once the SIE fizzles out Germany will have access to ~equally good AI. And if the industrial explosion has still got many years to go that could give them time to catch up.
There’s also the question of how Germany is able to use the US AI. For example, suppose two US projects are neck and neck. Or suppose there’s one US project, but multiple downstream ‘product’ companies that can sell fine-tune and sell access to the model. In that case, competition could lead for German companies to be able to access AI for a wide range of tasks. Including ‘R&D’ that really involves copying/adapting US tech developed elsewhere. In this case, Germany doesn’t have to use worse AI.
OTOH, maybe there’s just one US project and it’s single-handedly making tech progress in the US and selling its systems abroad. In this case, it could have strong incentive to prevent German companies using its AI to copy US tech. Bc it developed that tech itself! This is a case where the US is strongly coordinated bc all US economic activity is just one company.
Another scenario. There’s one US AGI project. It sells API access to existing companies in the US, at a very healthy profit, and those companies all make amazing new tech and own the IP. In this case, teh US project still has incentive to sell its AI to Germany companies that will use it to copy the tech developed in US companies. (In fact, there’s no reason the US companies would be ahead of the German companies in developing the tech in this scenario.) Here is seems like USG needs to step in to make sure US companies keep the profits.
Like, here’s one way of putting it. There’s def a chance that one project outgrows the world. Especially if there’s a monopoly on AI within the US with the support of the govt. That project would be internally coordinated and avoid trades that disadvantage it relative to RoW. But suppose one project doesn’t outgrow the world. There’s then a question of why the larger entity that does outgrow the world is the US. Like, why are other US companies in on the action, but no non-US companies. ASML and TSMC will have a lot to offer! My answer to this question is that USG would have to make specific efforts, significantly restricting trade, to make this happen.
Then the scenario where amazing tech is built in Germany by US companies. Let’s put aside manipulation and military exploitation for a second. The GDP produced by these companies is German GDP and the German government can tax it. So, if most physical manufacturing is not on US soil and most GDP is from physical stuff being produced, then >25% GDP will be produced on non-US soil and could be taxed by non-US governments. But yeah, i agree that the analogy of colonization is chilling here.
(sorry, this isnt’ very well organised. But overall my tentative position is that it is plausible US outgrows the world here, but it seems less certain and more complicated than what appears to be your position)

Tom Davidson 28 Aug 2025 18:47 UTC
6 points
0
on: Could one country outgrow the rest of the world?
Posting a brief exchange between myself and Sam Winter-Levy on this topic, relating to this piece of his.
Sam
Hi Tom,

Sam Winter-Levy here—I’m a fellow at the Carnegie Endowment for International Peace in DC. Really interesting piece on economic growth explosions. It overlaps with some work we’re doing at Carnegie on “decisive strategic advantages,” just wanted to share some quick reactions in case helpful!

I know you purposefully set aside military considerations, but I’m not sure I agree with the premise that outgrowing the world is a path to a “decisive strategic advantage without acting aggressively towards other countries.” In particular, I think that so long as nuclear deterrence and MAD hold, they are a binding obstacle to any alleged DSA, no matter how much AI turbocharges economic growth. Today, the US economy is 15x Russia’s and 1000x North Korea’s, but the US has no ability to impose “complete domination” over either of them, to put it mildly, nor would it even if AI dramatically boosted the US economy still further—in large part because of Russia and North Korea’s ability to credibly threaten to impose massive costs on the US through nuclear escalation on any issue of sufficient importance to either regime.

More broadly, I don’t think most claims about the existence of a DSA take nuclear deterrence seriously enough. Before nuclear weapons existed, a state’s ability to defend itself from coercion depended crucially on its economic size, since relative forces mattered and force size/quality depended ultimately on having a large population and an advanced economy. But that’s not true once you have a second-strike nuclear capability; nuclear weapons break the connection you emphasize between economic power and the ability to impose your political preferences on other nuclear-armed states. So I think even granting the argument that AGI could lead to some sort of economic explosion for one state, the amount of coercive leverage that would grant that state is likely to be highly limited, nowhere close to the Bostrom DSA definition you use, so long as nuclear deterrence remains intact.

In case of interest, here’s a piece we published recently on these issues, coauthored with Nikita Lalwani (former director for tech/nat sec on the Biden NSC). We make a version of the above argument about the importance of MAD for the existence of any alleged DSA, and then lay out how exactly AGI might threaten nuclear deterrence. Would welcome any feedback! We’re still tentatively thinking through these DSA issues ourselves.

Best,

Sam
Tom
Hi Sam,
Thanks for this great pushback!
I think you’re right that this is an oversight of the piece. But here are some responses:
1. In terms of controlling civilisation’s longrun future, outgrowing the world in the way we describe would be enough. We’ll eventually settle space and North Korea/Russia will be the tiniest fraction of total resources/‘land throughout space’. So if US controls everything but some countries on earth, they completely dominate the longrun future. Agree though the DSA definition i used is a bit different to that
2. If a country gains this kind of massive economic advantage, it will gain a massive tech advantage as well that, at some point, is likely to allow it to remove the second strike capability. This is more speculative, but we are here talking about the limits of technology and 1-month doubling nanobots etc. Still, this at least should be addressed by the post.

Tom Davidson 28 Aug 2025 18:17 UTC
2 points
0
in reply to: Daniel Kokotajlo’s comment on: Could one country outgrow the rest of the world?
Ah, cool.
I was roughly imagining:
- US gets to ASI first
- China 6 months behind
- Chinese AI used in Europe. US AI also used in Europe. Deep integration with the physical economies due to the massive complementarities.
So then the Q is why Europe can’t quickly use its AI to copy the new physical tech?
And similarly for China, which can use its worse AI (and the US AI as well if the status quo continues).
Also, even absent copying the new tech, the amazing new physical robots will be made on European soil. (But maybe US companies still capture the profits?)

Tom Davidson 28 Aug 2025 17:27 UTC
4 points
0
in reply to: Daniel Kokotajlo’s comment on: Could one country outgrow the rest of the world?
Thanks!
The key insight is that ordinary human leaking and spying and tech diffusion happens at ordinary human pace. If e.g. China is doing an intelligence explosion such that it’s making tech and economic progress at 100x normal speed, the process of ideas gradually diffusing to the US and causing catch-up growth won’t be 100x faster to compensate. Instead e.g. the lag between “the chinese are selling an exciting new product” and “our engineers have dissected it and deduced some interesting ideas about how it was made and replicated these ideas in our own manufacturing processes” will be… approximately the same? And so for practical purposes it might as well not be happening.
I don’t think it will be the same? If the economy is growing 100X as fast then that means:
- physical artefacts transported from place to place 100X faster
- new tech and products adopted throughout the economy 100X faster
Both these things help with diffusion between countries as well.
And if designing new tech is 100X faster, i’m not seeing why we would expect copying tech to be <100X faster. With copying, you start with additional clues—e.g. a physical artefact or using a digital product or public materials (like OAI’s o3 demo) -- and design the tech from there. Why would designing new tech from scratch be sped up by more than designing from clues? I.e. we’ve got super-smart AI doing both.
the rate of tech diffusion via leaks and spying will go way down
Yeah, agree spying will go way down once humans are replaced, and cyber hacks will go down too. Though i would have thought a small minority of tech diffusion comes from spies + hacks? E.g. i’d have thought selling and publishing about o1 was more important than those things, though i could be wrong and spies might be unually important for AI diffusion in particular.
...
I don’t think you’ve responded to my argument that companies in the leading country could make way more short-term revenues by producing and selling their technologies in other countries? (See the argument in the box.) Bc ASI cognitive capabilities are massively complementary with human labour and physical capital inputs.

Tom Davidson 28 Aug 2025 11:52 UTC
9 points
0
in reply to: Lukas Finnveden’s comment on: Could one country outgrow the rest of the world?
Why look at the number immediately after AGI, here, rather than further into the intelligence explosion?
Here was the thought:
If the lab hasn’t outgrown RoW by the time RoW develops AGI then we’re now in a position where RoW has more physical capital than the lab. This means that, in the next physical doubling, RoW will produce more physical capital than the lab, and so do more learning by doing than the lab has ever done. After that doubling, RoW will then have better tech than the lab and so their physical capital will double more quickly. So, after that doubling, RoW will have both more physical capital and their physical capital will double more quickly. This means they’re on a trajectory to always have more physical capital and the lab can’t outgrow them.
So this is implicitly assuming that the number of units produced is the hard bottleneck on the technology of physical capital, and the lab’s additional AI cognitive labour doesn’t help. This is in line with the classic economist view that you bump up against ultimate limits to what can be inferred from data. But I think qualitatively smarter AI could change this a lot. Maybe the lab has much smarter AI than RoW and so can learn much more per unit produced.
So, thinking about it more, I think this calc makes sense if we start the clock at the point when the SIE (software-only intelligence explosion) fizzles out. Because then, once RoW catches up, they’ll have ~equal cognitive inputs to the lab. So the lab really does need to overtake RoW on physical capital before this point. (Maybe the lab can get more chips than RoW? But this is a type of physical capital.)
Anyway, accounting for this makes it more plausible the lab can outgrow RoW. They need the 1 month doubling time robots not when they first have AGI, but when the software intelligence explosion has fizzled. Which means they have more powerful AI and longer to do robot experiments.
Another reason my calc understates the lab’s chance is that there may be higher value from experiments conducted in serial. RoW constructs all their physical capital in parallel in one doubling. Whereas the lab does it over multiple doublings one after the other. That could make a big difference.
How to do better?
The more nuanced way to game this out would be to represent, at each time-step, the physical capital, cognitive labour, and technology (for physical capital) of both the lab and RoW.
Then have a production function for how inputs of physical capital produced and cognitive labour improve technology in each time-step. Probably smg like g_A = (K^a * C^b)^lambda * A^-beta. Where a+b=1, lambda represents the benefits of doing research in serial, and beta accounts for ideas getting harder to find. (This accounts for your other point about the rate of production doubling every 3 doublings. The parameter beta controls whether it’s “3” or some other number.)
Then in each timestep simulate each actor’s change in:
- Physical capital
  - the level of technology determines the physical capital doubling time
  - so you combine current physical capital and technology to get new physical capital
- Technology
  - Apply the equation above for g_A.
- Cognitive labour.
  - Probably simplest to make this exogenous. Specify some trajectory given your views on the software-only intelligence explosion.
My overall view
- I think that physical capital doubling times of a few months shortly after AGI are more plausible than when I first wrote that appendix. So I think this is plausible if we’re starting the clock at the end of the SIE.
- Accounting improving serial experiments will favour the lab further. As will the possibility the lab can maintain a cognitive advantage beyond the SIE by designing and quickly manufacturing amazing chips.
- But the assumption that a company can spend $1trillion on physical capital before the end of the SIE still seems too high. Though it’s unrealistic to think RoW will use all their physical capital for growth, which balances this out.
- The assumption of a 9 month lead feels generous.
Maybe the most plausible scenarios involve ~6 month leads, ~$250b on physical capital, ~1 month robot doubling times at the end of the SIE, and doubling times becoming faster over time.
I made a simple of the dynamic here. (Still not modelling the gains from serial experiments and
So overall I think I’d still bet against the lab being in a position to do this on the economic fundamentals; but it is more plausible that I’d thought.
What links here?
- Could one country outgrow the rest of the world? by Tom Davidson (21 Aug 2025 15:32 UTC; 45 points)

Tom Davidson 28 Aug 2025 10:19 UTC
3 points
0
in reply to: Lorentz Manifold’s comment on: The Industrial Explosion
Thanks for this helpful comment.
Some quick follow-up Qs.
First:
Of course, scale does have an effect; larger animals have slower metabolisms (smaller specific surface area, difficulty with heat dissipation). However, you can see that there is no order-of-magnitude difference in metabolic rate between a hummingbird and an elephant, so this is not the primary contributing factor.
GPT-5 thought that the ratio of metabolic rates was 100X. Two OOMs!
Second, you say that nanotech could only replicate very fast if it harvests the ready-made compounds in the biosphere. But plant cells that make their food from photosynthesis can replicate in hours/days. E.g. duck weed.
Third, I’d be interested if you had sources for your claims against J/kg and W/kg

Tom Davidson 28 Aug 2025 10:04 UTC
2 points
0
in reply to: David Duvenaud’s comment on: Thoughts on Gradual Disempowerment
In the absolute Malthusian limit AI won’t earn more than its running costs and can’t save. So if we expect AI wages to approach that limit, that seems like a strong reason not to expect that humans will keep more money than AI

But yeah, wages probably won’t be completely up against that limit in practice, as for humans throughout history. But i think it might be pretty close

Tom Davidson 28 Aug 2025 9:54 UTC
2 points
0
in reply to: David Scott Krueger (formerly: capybaralet)’s comment on: Thoughts on Gradual Disempowerment
Thanks!
It feels like this statement should be qualified more… this is holding GD to a higher bar
Yeah fair. I’ve edited to qualify it more.

The analysis of economic disempowerment seems to take place in a vacuum, ignoring one of the main arguments we make, which is that different forms of disempowerment can mutually reinforce each other. The most concerning version of this, I think, is not just “we don’t get UBI”, but rather that the memes that say “it’s good to hand over as much power as quickly as possible to AI” win the day.
Yeah it’s a bit tricky to know how to structure the argument when you’re saying that the 3 domains all mutually reinforce each other. Like, after reading the paper it was unclear to me why the 3 domains don’t mutually reinforce each other to remain good given that they start good. The order in which the paper’s sections appear, and the arguments within, suggested that the main mechanism was:
- Humans lose significant economic power
- Then they start losing cultural power and influence over the state
- Then they have a small amount of each that they ultimately are disempowered in all domains
And so a natural response (which i gave) is:
- Humans will keep economic power
- As a result, they’ll keep cultural influence and control of the state
But yeah, i agree cultural shifts to favour hand off will happen even absent economic disempowerment. I think just driven by ordinary economic competition. And if we hand off to sufficiently misaligned AI, we’re screwed. Assuming AI is aligned enough that it never seeeks power, i’m not sure how worried we should be about handoff. But plausibly we should demand a higher alignment bar than that.
Anyway, to my mind this argument is better understood as “competitive pressure to hand off to misaligned AI” than as an interplay between economic and cultural and state disempowerment, but I do buy it.

The analysis of cultural disempowerment goes one step “worse”, arguing that “If humans remain economically empowered (in the sense of having much more money than AI), I think they will likely remain culturally empowered.” I think we agree that a reasonable model here is one where cultural and economic are tightly coupled, but I don’t see why that means they won’t both go off the rails. You seem to think that they are almost guaranteed to feedback on each other in a way that maintains human power, but I think it can easily go the opposite way.
Yeah, to clarify, i don’t feel it’s guaranteed to maintain human power. Overall, I feel like “yeah i guess maybe that could happen, though none of the mechanisms you mention seem that convincing and there seem like counter considerations and humans will have a strong incentive to keep power if they can and (hopefully!) truthful AI advice to help them and myopically aligned AIs to implement things to help… and also we can see this playing out in real time and respond so not sure it’s worth focussing on in advance (though i agree it will be worth focussing on while it’s happening)”.
Do you think that, absent AI power-seeking, this dynamic is highly likely to lead to human disempowerment? (If so, then i disagree.)
Regarding political disempowerment, you state: “It’s hard to see how those leading the state and the top AI companies could be disempowered, absent misalignment.” Personally, I find this quite easy. Insufficient elite coordination is one mechanism (discussed below). But reality can also just be unfriendly to you and force you to make choices about how you prioritize long-term vs. short-term objectives, leading people to accept deals like: “I’ll be rich and powerful for the next hundred years, and then my AI will take over my domain and do as it pleases”. Furthermore, if more people take such deals, this creates pressure for others to do so as well, since you need to get power in the short-term in order to remain “solvent” in the long term, even if you aren’t myopic yourself. I think this is already happening; the AI arms race is burning the commons every day; I don’t expect it to stop.

I said “absent misalignemnt”, and I think your story involves misalignment? Otherwise the human could hand off to AI that represents their interests. Clearly there’s a problem with handoff if AI seeks power. And i agree it seems bad if AI doesn’t seek power that but also won’t represent human interests as it governs. Though i feel a bit confused about how humans are never able to coordinate to reign it all back in if AIs aren’t seeking power.

Tom Davidson 28 Aug 2025 9:18 UTC
2 points
0
in reply to: Lukas Finnveden’s comment on: Thoughts on Gradual Disempowerment
Overall, i think this considerations favours economic power concentration among the humans who are legally allowed to invest in the most promising opportunities and have AI advisors to help them
And, conversely, this would would decrease the economic influence of other humans and AIs

Tom Davidson 28 Aug 2025 9:14 UTC
2 points
0
in reply to: Lukas Finnveden’s comment on: Thoughts on Gradual Disempowerment
Thanks—yep that’s what i meant!

Tom Davidson 20 Aug 2025 12:45 UTC
5 points
1
in reply to: Ted Sanders’s comment on: The Industrial Explosion
Yeah I think you’re right. Good spot.

I’d be interested in whether Epoch think this slower rate is the better one to extrapolate given that they tend to use the faster rate in their headline figures

Tom Davidson 19 Aug 2025 13:54 UTC
2 points
0
in reply to: David Duvenaud’s comment on: Thoughts on Gradual Disempowerment
Why does it matter whether AIs’ wages are subsistence-level? This seems to prove too much, e.g. “monkeys won’t be threatened by human domination of the economy, since the humans will just reproduce until they’re at subsistence level.”
I think it matters bc AIs won’t be able to save any money. They’ll spend all their wages renting compute to run themselves on. So it blocks problems that stem from AI having more disposal income and therefore weighing heavily on economic demand signals.
But perhaps you’re worried about other problems. E.g. the economy might still be geared towards creating and running AI chips, even if AI lacks disposal income. But the hope would be that, if humans control all disposal income, then the economy is only geared towards chips to the extent that’s instrumentally helpful for humans.
Re monkeys, a disanalogy is that none of the human economy is geared towards providing monkeys with goods and services. Whereas that will be the case with AI.
form a legal super-organization and blur the lines between consumption, trade, employment, and capex
Interesting idea that AI systems could blur the lines between their own consumption and “spending that’s instrumentally useful for producing goods and services”. That does seem like a distinctive and interesting way to tip the economy towards AI preferences, and one that would dodge tax. (Though it’s a problem even if AIs aren’t paid wages. To the extent that AIs are paid wages and are thereby economically enmpowered, taxation still seems like a good solution to me.)

Tom Davidson 19 Aug 2025 13:46 UTC
2 points
0
in reply to: David Duvenaud’s comment on: Thoughts on Gradual Disempowerment
Thanks, these are interesting ideas though i’m not quite seeing the connection to my point.
I was imagining some fraction of ppl want to use all their current wealth to bring about a good long-run future. Strategy: they invest it on the market and eventually spend the proceeds on whatever is best. They don’t reproduce (as that would cost money, which they want to invest).
Maybe they upload themselves to live indefinitely if that’s needed to know how to best spend the money, but it will be cheap. Tbh, even staying alive indefinitely in biological form will likely be very cheap in absolute terms for someone with decent saving today.
Malthusian dynamics don’t seem to block this strategy
I agree Malthusian dynamics are important to think about post-AGI, and like your examples of ways that you could technically ‘keep humans alive’ more cheaply. Though i’m not sure if you fully appreciate how cheap it would be to keep all current humans around and limit reproduction to 8 babies per person. (Longer term you’ll need stricter limits.)

Tom Davidson 19 Aug 2025 13:35 UTC
3 points
0
in reply to: David Duvenaud’s comment on: Thoughts on Gradual Disempowerment
Thanks for linking to that comment—great stuff.

I think this might be a case where ‘absolute disempowerment’ and ‘everyone dying’ comes apart from ‘relative disempowerment’ and ‘we get a much worse future than we could have done’. Seems more plausible AI advisors don’t sufficiently forewarn about the latter

Tom Davidson 19 Aug 2025 13:25 UTC
4 points
0
in reply to: Jan_Kulveit’s comment on: Thoughts on Gradual Disempowerment
Thanks!
Appreciate the many concrete examples you’re giving here.
Responding quickly.
I do not think the convincing versions of gradual disempowerment either rely on misalignment or result power concentration among humans for relatively common meaning of alignment roughly at the level “does what the developer wants and approves, resolving conflicts between their wants in a way which is not egregiously bad”. If “aligned” means something at the level “implements coherent extrapolated volition of humanity” or “solves AI safety” than yes.
Yep, that makes sense. And I disagree, so this is useful clarification.
I think that if AI “does what the developer wants and approves, resolving conflicts between their wants in a way which is not egregiously bad” then I am much less worried about GD than about power-seeking AI. (Though i have some uncertainty here if the AI is resolving these conflicts pretty badly but hiding the fact it’s doing this for some reason. But if it’s resolving these conflicts as well as fairly competent human would, i feel much less worried about GD than powre-seeking AI.)
----- difficulty of indexing economy undergoing radical technological transiton (as explained in an excellent post by Beren we reference)

It’s more than just index funds. It’s ppl getting AIs to invest on their behalf, just like VCs invest on ppl’s behalf today. It seems like we need fairly egregious misalignment for this to fail, no?
problems with stability of property rights: people in the US or UK often perceive them as very stable, but they depend on state enforcing them → state becomes a more load-bearing component of the system
Why is it more load bearing than today? Today it’s completely load bearing right? If income switches from wages to capital income, why does it become more load bearing? (I agree it becomes more load bearing when taxation is needed for ppl’s income—but many ppl will own capital so not need this)
having overwhelming cultural production power means setting consumer preference
Thanks, interesting point. Though humans will own/control the AIs producing culture, so they will still control this determinant of human preferences.
right-wing US twitter discourse is often influenced by anonymous accounts run by citizens of India and Pakistan; people running these accounts often have close to zero econ power, and their main source of income is the money they get for posts
Interesting. And you’re thinking that the analogy is that AIs will have no money but could have a big cultural influence? Makes sense. (Though again, those AIs will be owned/controlled by humans, somewhat breaking the analogy.)
Again, consider Musk
But the ideas that are bad for Musk and his thinking have generally decreased his power + influence, no? Overall he’s an exceptionally productive and competent person. If some cultural meme caused him to be constantly addicted to his phone, that wouldn’t be selected for culturally.
we assume institutional AIs will be aligned to institutions and institutional interests, not their nominal human representatives or principals
So what causes the govt AIs to be aligned to the state over the heads of office, to the extent where they disempower those humans? Why don’t those humans see it coming and adjust the AI’s goals? Or, if the AI is aligned to the state, why doesn’t it pursue the formal goals of the state like protecting it’s ppl?

Tom Davidson 15 Aug 2025 9:51 UTC
2 points
0
in reply to: dr_s’s comment on: How quick and big would a software intelligence explosion be?
I agree with the general point, but don’t think it applies to this model.
I’m not calculating anything to a high degree of precision, inputs or outputs.
There aren’t complicated interaction effects with lots of noisy inputs such that the model might overfit to noise.
I could have dropped the code, but then i’d have a worse understanding of what my best-guess inputs imply about the output. And it the analysis would be less transparent. And other couldn’t run it for their preferred inputs.