How can the middle powers avoid getting trounced during the intelligence explosion? A plan.
Superintelligence will likely be developed by US companies; run on US datacentres; and be under the jurisdiction of the US government. This will massively boost US’ military power and make the US economically dominant (eg US producing 99% of world GDP). By default, middle powers will be left in the dust.
How can middle powers avoid this fate? It’s tough, but here’s the best plan I could think of. (I’m particularly thinking about liberal democracies with influence over AI like UK, Europe, Japan, South Korea, Taiwan.)
On a very high-level: middle powers should leverage the fact that the US needs them to beat China. It’s genuinely unclear which country will develop superintelligence first, and which would win in a subsequent industrial explosion. Middle powers should help the US,and make sure they are rewarded with continued access to frontier AI and new technologies (including military tech).
That final boded part is hard. What can the UK realistically do if the US denies it access to frontier AI? The middle powers need a credible alternative to being supplicants of the US. The only alternative that makes sense to me is siding with China. If the US won’t grant middle powers access to their frontier AI, but China will, why should middle powers continue to send AI chips to the US? Why should they continue to support the US diplomatically and militarily? They shouldn’t. They should be willing to pivot to China if the US doesn’t offer AI access sufficient for their national security needs.
My plan for the middle powers has two stages:
Maintain as much economic and military leverage as possible during the intelligence explosion.
Use that leverage to ensure that, when superintelligence is developed, it refuses to help the US (/China) disempower the middle powers.
Stage 1 could well be enough by itself. Maybe middle powers can maintain significant economic and military power indefinitely. But if not, stage 2 is a back-up: it binds the US so that it can’t use its dominance to crush the middle powers.
I’ll walk through each stage in turn.
Stage 1: Maintain as much economic and military leverage as possible during the intelligence explosion
The biggest lever here is securing access to frontier AI. Anton Leicht has a great post about how this is under threat, as evidenced by developments with Mythos. Middle powers should insist on equal commercial terms to US companies, and comparable access for their militaries. This is in AI companies’ interests! A bigger market means more customers and higher prices.
Aside: why access to frontier AI might be sufficient for middle powers to stay economically relevant indefinitely
The hope here is that:
Most of the economic surplus from AI is not captured by AI companies. To create economic value, AI must be combined with complementary inputs: factories, human physical labour, know-how of human experts, relationships with suppliers, trusted brands, etc. How much of the surplus will be captured by AI companies vs the owners of these complementary inputs? Optimistically: producers of general-purpose technologies often capture only a small fraction of surplus; and multiple frontier AI companies might sell similar products and bid each other down on cost.
Most of the economic surplus from AI occurs outside the US. The majority of these complementary inputs are situated outside the US. So most AI-driven economic value-add should occur outside US borders.
If (1) and (2) both hold, a significant fraction of AI’s economic surplus will accrue to non-US actors.
But how can middle powers guarantee frontier AI access? It’s tough, but a few strategies:
Build data centres. Partner with frontier AI companies to build secure data centres domestically, in return for guaranteed frontier access. This is a big win-win. AI companies improve their bargaining position with the US government. Recall, the US government threatened to destroy Anthropic when Anthropic insisted that their AI systems wouldn’t be used for legal mass surveillance.
Adopt AI. The more middle powers use frontier AI, the more costly it is for AI companies to cut them off.
Invest in frontier AI companies. Once they IPO, middle powers could invest billions or trillions into leading AI companies, in return for access guarantees.
Support the US internationally. If middle powers throw their diplomatic and military weight behind US foreign policy objectives, it benefits the US to keep them strong.
Build a relationship with China. If the US refuses to grant middle powers access to frontier AI, the national security implications are dire. Middle powers need a plan B, and China is the only other game in town for frontier AI. Only if this alternative is truly credible can it be leveraged into access to US frontier AI.
Ultimately, this involves middle powers threatening to sell semiconductor equipment and chips to China instead of the US. Obviously, that’s pretty far outside the Overton window. But that may change as the world rapidly wakes up to powerful AI and its national security implications.
Demand kill switches on US data centres. This is much more late-stage, after the world has truly woken up to the strategic implications of AGI. Suppose US and middle powers agree to a “chips for frontier access” deal – middle powers continue to supply the US with frontier chips; US continues to give middle powers access to frontier AI. The middle powers might still worry: what if the US suddenly changes its mind once it has superintelligence? By then, the US might be powerful enough to dominate without continued allied support. This is where kill switches can help. If the US withdraws AI access, allies could destroy US data centres in response. It’s a way to lock-in the deal.
(h/t AI futures project for this idea. A related idea is for US data centres to be placed in a location that’s easy to attack – like in space)
Beyond securing access to frontier AI, how else can middle powers maintain economic and military leverage?
Build physical infrastructure. Factories, robots, solar panels, batteries, semiconductors — all these industries are highly complementary to powerful AI.
Maintain nuclear 2nd strike capability. The point isn’t to use it. But it improves their leverage for stage 2.
The catch-all meta-point here is waking middle powers up to superintelligence.
I’m not recommending middle powers do their own frontier AI development. Seems very hard for them to catch up with the US.
Stage 2: Ensure that, when superintelligence is developed, it refuses to crush middle powers
If stage 1 goes well, middle powers remain somewhat powerful economically and militarily deep into the singularity. But it might fail. What can middle powers do if they see the US on track to total global dominance?
First, they should demand a pause/slowdown of AI development. But the US may refuse – pausing is very costly if alignment risk is low. And pausing is a stopgap: eventually, superintelligence will be developed.
An additional demand: when superintelligence is developed, it’s designed to refuse to crush middle powers. By doing this, the US would credibly bind itself to maintaining the sovereignty of other nations.
The optimistic case here is that this isn’t a big sacrifice for the US. They can still become as rich as they like and achieve their security interests. Sure, they can’t seize control of other nations, but that is not an important goal of theirs anyway. Losing that option is well worth the benefits: other nations cooperate economically, don’t attack US data centres, and don’t threaten nuclear war.
The pessimistic case is that this involves an insane degree of irrevocable hand-off to AI. The US must literally be unable to attack middle powers no matter how hard it tries: retraining the AI, turning it off, training a new more powerful AI, passing new laws, using the military to destroy the datacentres the AI is running on. For it to be truly binding, the US must permanently hand over military and political power to AI. That might be deeply unpopular, and indeed seem insane to the US. It’s also very hard to verify: you can’t just verify the training run, you need to verify that humans+other AIs have no way to disempower the trained AI. It’s more like verifying “who would win this civil war” than “technical property XYZ holds”.
The realistic path here probably involves gradually handing off more and more control to AI that refuses to crush middle powers, with no clear point at which humans could no longer wrest back control.
The longer middle powers wait to push for stage 2, the less leverage they will have because the US will have pulled further ahead economically and militarily. So they should be pushing in this direction constantly. Eg demanding transparency into the model specs of powerful AIs deployed in the US government, and arguing that powerful military AI should be designed to obey international law.
(I described the plan as involving two stages because that’s how I expect it to play out over time. But succeeding at either stage is sufficient! If middle powers stay economically/militarily competitive, they never need to bind US superintelligence. And if they do bind superintelligence, they won’t be crushed no matter how far behind they fall.
Is it good to avoid middle powers getting trounced?
I live in the UK, so I am biased here. I do not want the UK to become a supplicant to the US!
But here’s a brainstorm of pros and cons from a more impartial perspective.
Pros to empowering middle powers:
Avoid a single point of failure. If the US becomes globally dominant and its political system fails, that’s a global failure.
More democracies. Many middle power democracies look more robust than the US, so more middle powers may mean more democracy.
Improve the US. Middle powers will have an interest in maintaining free market democracy in the US. “Free market” bc they’ll want multiple AI companies competing to sell cheap API access to non-US countries. “Democracy” because they’ll expect that the US is more likely to maintain a strong alliance with middle power democracies if it stays democratic.
Experimentation. Experimenting with multiple different political and legal systems seems generally good for figuring out a good way to govern society post AGI.
Pause AI. They could potentially pressure US/China to pause/slowdown reckless AI development.
Prosocial norms. When multiple actors bargain with each other (e.g. about how to distribute space resources, whether to develop a dangerous technology), they tend to frame arguments in terms of pro-social norms, and so agreements tend to emphasise the actor’s more virtuous/ethical values.
Cons of empowering middle powers. Multipolarity has its own downsides:
more likely to lead to war.
can drive extreme competition, eg racing to develop a dangerous technology, or to hand off power to misaligned AI
Harder to prevent harms from offense-dominant technologies like bioweapons
my plan involves waking up middle powers, which could shorten timelines
I think that “middle power” is a bit too broad and too narrow for the question you’re asking. For example, on the “too broad” front:
There’s Iran, which is friendly with Russia and China, and certainly gets military support from them, but also infamously tried to compete with Russia for control over Syria in a way that ultimately compromised Syria’s ability to remain stable. It’s sovereign, culturally distinct from, and can occasionally pick outright fights with its allies, but it’s not in the same weight class as them.
There’s France, which has its own military, its own (non-frontier) LLM, and domestic industry, but America can (infamously, around five years ago) straight up veto their arms export deals when they become inconvenient, and, whether we like it or not, American culture is pretty dominant in France. French anti-US sentiment is essentially indistinct from the complaints of US liberals, for example.
There’s India, which doesn’t really have one patron, and punches below its weight militarily[1], but does have a large economy. Their relationship with China is adversarial, and their reputation in America is declining quickly, so their advantages/disadvantages are very different from those of the two countries above.
If an intelligence explosion is defined as a model’s capabilities increasing vertically, faster than humans can react, then it’s very likely that America/China are both much too large that even treating them as unified beneficiaries is plausible. If the model has its own “wants”, then the model is now in charge. If the model has been successfully aligned to answer to some group of people, then that group is now in charge. The window in which a model is sufficiently broad in its allegiance that it believes it is loyal to “all Americans”[2], but sufficiently narrow that it doesn’t care about anyone else, seems extremely small, and I don’t think that anyone—even the most fervent American patriot—is aiming for it.
If you define intelligence explosion more broadly, as your post appears to indicate, then I’d argue that “middle powers” get on as they’ve been getting on already. The benefits of the Russo-Chinese AI-industrial complex flow to Iran because they want Iran to survive, and Iran has leverage because these countries, again, want Iran to survive in its strategic position. Likewise, negotiations of terms aside, the EU essentially pushes in the same direction as the US on most if not all international matters, and while its domestic industry would be hollowed out significantly by a hypothetical leap in AI performance that America declined to share out, the products of American industry would definitely continue to be exported.
Seeing lackluster results in a recent conflict with Pakistan, which is six times less populous and doesn’t have nearly the same number of Western/Eastern arms suppliers
How do you define that, anyways? Is a guy who showed up on a boat one day ago American, even if he doesn’t care about America, doesn’t speak the language, and has no generational ties? If that’s the case, then you can just reclassify the entire planet as citizens and the issue’s solved.
Superintelligence will likely be developed by US companies; run on US datacentres; and be under the jurisdiction of the US government. This will massively boost US’ military power and make the US economically dominant (eg US producing 99% of world GDP). By default, middle powers will be left in the dust.
How can middle powers avoid this fate?
If ASI is developed, they have a decent chance of avoiding this fate due to extinction from ASI misalignment or misuse, which imo is a much worse fate.
And pausing is a stopgap: eventually, superintelligence will be developed.
if we wanted to and kept wanting to, we could just ban AGI for a very long time and maybe forever.[1]instead of making non-human-descended top thinkers, we could just continue carefully becoming more capable and intelligent as humans. i don’t think this would be that weird. regulation of development (and specifically the development of intelligence) much more stringent than a very long AGI ban is probably going to happen in any future with intelligent life, because probably many aspects of anyone’s values cannot compete/[remain in control]/[be realized]/survive through that much unregulated development. (however, this stringent control end up mostly being self-regulation of a singleton.) that is, extreme regulation of the development of thought is probably inevitable[2]; there’s just a question of whether it’s implemented by humans or some future AI(s).
that said, i agree it is unlikely that we ban AGI in practice (i think it’s likely we make an AGI this century and that’s the worst thing ever done). i’m in part writing the present comment because a nontrivial contributor to it being unlikely that we get an AGI ban is that people think and say this sort of stuff. i’ve been irked by the same sort of fatalism about AGI in a bunch of other writing by forethought, 80000 hours, and many others in EA / AI safety and beyond. to clarify: i’m certainly not asking anyone who thinks an AGI ban is unlikely to lie about that, and i’m also not asking anyone to stop saying “an AGI ban is unlikely” when that is pertinent. but i think many people are weirdly systematically speaking as if an AGI ban is not an option worth considering, including many who in fact believe that it would be a good idea to ban AGI for a long time (if it could be done) or at least have significant probability on that
i think this is also a better thing for middle powers to fight for than what you propose (which i’d semi-seriously call a collaborationism-maxing policy package), for the sake of the world and also for their own sakes
ok maybe a landian/marxist/[nature red in tooth and claw] scenario where everything that isn’t competitive keeps getting annihilated is also possible, idk. tbh i just really wanted to state this antithesis to “AGI is inevitable”. maybe some qualifier like “if the future is not absolutely ruthless” needs to be added.
I have a hard time piecing together the context and the target of this plan. Is this plan designed to be convincing to certain people, vs. what are your beliefs about the world that underly it? (I don’t know who you are; if your views are somewhat known around here, sorry about that, I missed it.) You talk at the same time about superintelligence taken seriously, but then say that it will be built, and suggest (maybe, not clear on this point) that countries help it being developed. This looks plainly insane to me; but maybe you mean that countries won’t take superintelligence seriously, or are run by cursed madmen, so the plan you design is a complex plan that takes into account their wants and beliefs to feed them something shovel-ready that increases the likelihood they’ll do something better on your own terms, or are you also a madman that will not just stop pursuing superintelligence? I mostly agree with what I guess is the MIRI view here, that if I am another country and I believe in upper-case-S superintelligence, then if the US wants to go and build it what I do is start a nuclear weapon program to make it clear what the payoffs are.
I feel quite nervous about this plan (primarily “middle powers threatening to sell semiconductor equipment and chips to China instead of the US”).
Navigating the intelligence explosion feels hard enough as is. I’m worried that encouraging middle powers to play both sides will just lead to chaos.
I see where you’re coming from. The US is much less trustworthy than it used to be. I can imagine worlds where this plan would lead to good outcomes. However, my intuition is that the chances of this plan blowing up in our faces is way too high. Maybe, I’d change my view if I spent more time reflecting and digging into the facts, but this is where I am at the moment.
”For it to be truly binding, the US must permanently hand over military and political power to AI.”
Also, this is not something we want to rush. I struggle to see how this could happen fast enough to reassure the middle powers, whilst also being late enough that this wouldn’t be utterly reckless. The best counter I was able to come up with is that being able to counter threats might force a handoff anyway and that there might counterintuitively be no difference in the amount of trust you need for a temporary handoff vs. a permanent handoff given the ability of an AI to transform the former into the later.
If we’re in a sim, it’s being used for acausal trade
Me: Our world is exactly the kind of thing you’d simulate if you were doing acausal trade! It’s just before civilisation develops the ability to lock-in deals.
Sceptic: Sure, but there’s other reasons ppl might simulate earth. Maybe it’s for ppl’s entertainment? Maybe it’s social science, exploring alternate histories?
Me: For sure. But whatever the purpose of the sim is, it will contain info that’s relevant to ppl that want to do acausal trades. It will have info about who has power post-AGI, what their values are, and whether they want to do acausal trade. If someone ran the sim for entertainment, they’d obviously sell that info to the acausal trade folks.
Sceptic: Won’t the acausal trade folks just run their own sims?
Me: Maybe! But they’ll be keen to buy relevant info from others who runs sims. If others run earth sims for entertainment, the acausal trade folks will buy the info and run fewer earth sims themselves.
At this point we can’t be very sure that acausal trade is even a thing, since we don’t have a formal solution to decision theory we can be confident in (or even meet the lower bar of not having clearly serious flaws), and therefore can’t derive acausal trade from it as a solution to some decision problem. (Or have some other way to gain high justified confidence that acausal trade is actually a thing.)
And then even if acausal trade is a thing, whoever is running sims of us may not be running a decision theory that allows or recommends acausal trade, or we may not have anything to trade that they want, or the trade fails for some other reason like high “transaction costs”.
I was reacting to the lack of hedging or explicit uncertainty in the first line “If we’re in a sim, it’s being used for acausal trade”. Possibly Tom didn’t mean to express very high confidence by it (I see that in a more formal occasion he does explicitly state his uncertainty about acausal trade[1]), but I feel an obligation to err on the side of caution here, in case some readers do interpret it as expressing high confidence.
Quote from the link: Of course, unlike the causal case, whether consensus goods get funded depends on whether agents want to do acausal cooperation at all—which depends on their decision theories and their beliefs about their degree of correlation with others.
Ah, cool, yeah, fair enough. I interpreted that as more of a title, and am used to titles usually skipping epistemic qualifiers because you really want them to be short.
But I wouldn’t rule out someone who’s thought a lot about DT being pretty confident. I don’t think you need to need “solve” DT to be v confident that acausal trade is a thing anymore than you need to solve ethics to know that murder is wrong.
I could imagine that some of the acausal trade crowd have thought long enough about the space of decision theories and their implications to conclude that acausal trade is a consequence of many plausible DTs and is very likely happening.
My understanding is that even with CDT you can get sim-based trade (which i’d consider a form of acausal trade), and that on a first pass EDT and UDT both imply that acausal trade makes sense. So we only need some powerful agents to do one of these decision theories for acausal trade to go ahead.
I guess I can imagine a countercase like “bc of threats very few civs do acausal trade”, though it’s hard to see it go down to zero. I’d be curious if you have other counter-cases in mind.
(In general i’d defer to someone who thinks about this more on how likely acausal trade is to happen overall!)
I could imagine that some of the acausal trade crowd have thought long enough about the space of decision theories and their implications to conclude that acausal trade is a consequence of many plausible DTs and is very likely happening.
I’m not aware of anyone who has claimed this, and wouldn’t trust such claims anyway, since humans just aren’t that good at fully delineating the space of plausible solutions to some philosophical question. E.g. for the first half century of decision theory as a field, nobody thought that updatelessness might be worth investigating.
(In general i’d defer to someone who thinks about this more on how likely acausal trade is to happen overall!)
I wouldn’t defer a lot, because humans are also bad at finding flaws in their favorite philosophical ideas. Think of, e.g., theorists/proponents of Objectivism or Communism.
I’d be curious if you have other counter-cases in mind.
There are too many civilizations in the multiverse to simulate all of them. We can only do a sampling and then decide how to trade based on the statistical properties, but this gives an incentive to free-ride (i.e. getting the benefits of trade from other civilizations that did not specifically simulate us, without paying the costs), which may cause an overall breakdown in trade.
There are too many civilizations in the multiverse to simulate all of them. We can only do a sampling and then decide how to trade based on the statistical properties, but this gives an incentive to free-ride (i.e. getting the benefits of trade from other civilizations that did not specifically simulate us, without paying the costs), which may cause an overall breakdown in trade.
Hmm, but if each civ simulates 1 million others, every civ should be simulated 1 million times? I.e. the more civs in the universe, the more civs to do the simulating. And then i think there’s a stable equ where you only give nice things to the civs you simulate if you see them giving nice things to the ones they simulate. I suppose there’s a q of whether we’re able to achieve that equ?
Also, you can do the thing where when your sims run their own sims, you insert your universe into their sim, guaranteeing 2-way trade?
tiny nit: non-trade sims could happen late in the universe when the available lightcode isn’t big enough to impact trade much, or people could have already locked in their acausal trade decisions, or something analogous in another universe.
But then, entertainment sims might oversample entertaining outcomes (e.g. rerun an event 10 times and use the branch with the most entertaining result). And then, as a result, acausal trade folks’ marginal willingness to pay for info about how those entertaining worlds turn out would go down. And for similar reasons, the stakes of our actions would then generally be lower in any given entertainment sim.
If someone ran the sim for entertainment, they’d obviously sell that info to the acausal trade folks
Weak argument? The set of times that people incidentally produce information relevant to other people is much broader than the set of times they sell it to them.
I think I mostly agree with what this is actually saying, but I’m not sure it’s definitively acausal trade. e.g. it might be for just mundane trade, dividing gains by simulating Shapley values or something as part of a (very elaborate) cooperative bargain.
You can read about what it’s like to work with us here.
We’re currently hiring researchers, and I’d love LW readers to apply.
If you like writing and reading LessWrong, I think you might also enjoy working at Forethought.
I joined Forethought a year ago, and it’s been pretty transformative for my research. I get lots of feedback on my research and great collaboration opportunities.
The median views of our staff are often different from the median views of LW. E.g. we probably have a lower probability on AI takeover (though I’m still >10% on that). That’s part of the reason i’m excited for LW readers to apply. I think a great way to make intellectual progress is via debate. So we want to hire ppl who strongly disagree with us, and have their own perspectives on what’s going on in AI.
We’ve also got a referral bounty of £10,000 for counterfactual recommendations for successful Senior Research Fellow hires, and £5,000 for Research Fellows.
The deadline for applications is Sunday 2nd November. Happy to answer questions!
When I speak to ppl from DC, I’m told that the government and military will be very slow to adopt new tech.
If these two things are both true, there’s a scary implication.
If tech progress speeds up by 30x relative to recent history, then a 3-year procurement delay by the military means they’re deploying tech that’s effectively 100 years outdated. Even a 1-year delay at that pace means your military is fielding equipment from a completely different technological era. The US military spends ~$1T/year. But with tech that’s 30–100 years more advanced, you could potentially defeat them at 1/100th of the spending — just $10B!
A rogue actor — a private company or a government clique bypassing standard procurement — could spend a tiny fraction of the official military budget on cutting-edge tech and potentially overmatch the entire conventional military.
There’s amassive untapped potential for cheap military dominance that no legitimate actor will exploit bc only the military has legitimate authority to procure weapons and they are bureaucratic.
Here’s a visualisation of the basic dynamic:
Possible solutions:
Military procurement needs to get dramatically faster during an intelligence explosion. New AI systems and AI-produced technologies need to be rapidly integrated into official military capabilities. This probably means automating the procurement process itself. This is counterintuitive from some AI safety perspectives — many people’s instinct is to delay military AI deployment.
Delay rapid tech progress until military procurement has become super fast + safe. This might in practice involve delaying the intelligence explosion and/or the industrial explosion as well
Better monitoring/surveillance to make sure no one is secretly building military tech
If everyone in our universe doing acausal trade coordinates, we can sell “cosmic real estate” for monopoly prices
Let’s assume that there are many different universes (or Everett branches) that acausally trade.
Some traders won’t about “resources in our civ’s future lightcone” linearly. As a toy example, the leader of a distant alien civilisation might want to get a statue of themselves in as many different other universes as possible.
If many different actors in our universe do acausal trade, and compete with each other to trade with the alien leader, then they’d bid down the price of building that statue. Whereas if they all band together, they could hold out for a much higher price. So it could be in our collective interests to coordinate and “price fix”.
This is an example of a civilisation-wide public good that could be important long into the future.
How can the middle powers avoid getting trounced during the intelligence explosion? A plan.
Superintelligence will likely be developed by US companies; run on US datacentres; and be under the jurisdiction of the US government. This will massively boost US’ military power and make the US economically dominant (eg US producing 99% of world GDP). By default, middle powers will be left in the dust.
How can middle powers avoid this fate? It’s tough, but here’s the best plan I could think of. (I’m particularly thinking about liberal democracies with influence over AI like UK, Europe, Japan, South Korea, Taiwan.)
On a very high-level: middle powers should leverage the fact that the US needs them to beat China. It’s genuinely unclear which country will develop superintelligence first, and which would win in a subsequent industrial explosion. Middle powers should help the US, and make sure they are rewarded with continued access to frontier AI and new technologies (including military tech).
That final boded part is hard. What can the UK realistically do if the US denies it access to frontier AI? The middle powers need a credible alternative to being supplicants of the US. The only alternative that makes sense to me is siding with China. If the US won’t grant middle powers access to their frontier AI, but China will, why should middle powers continue to send AI chips to the US? Why should they continue to support the US diplomatically and militarily? They shouldn’t. They should be willing to pivot to China if the US doesn’t offer AI access sufficient for their national security needs.
My plan for the middle powers has two stages:
Maintain as much economic and military leverage as possible during the intelligence explosion.
Use that leverage to ensure that, when superintelligence is developed, it refuses to help the US (/China) disempower the middle powers.
Stage 1 could well be enough by itself. Maybe middle powers can maintain significant economic and military power indefinitely. But if not, stage 2 is a back-up: it binds the US so that it can’t use its dominance to crush the middle powers.
I’ll walk through each stage in turn.
Stage 1: Maintain as much economic and military leverage as possible during the intelligence explosion
The biggest lever here is securing access to frontier AI. Anton Leicht has a great post about how this is under threat, as evidenced by developments with Mythos. Middle powers should insist on equal commercial terms to US companies, and comparable access for their militaries. This is in AI companies’ interests! A bigger market means more customers and higher prices.
Aside: why access to frontier AI might be sufficient for middle powers to stay economically relevant indefinitely
The hope here is that:
Most of the economic surplus from AI is not captured by AI companies. To create economic value, AI must be combined with complementary inputs: factories, human physical labour, know-how of human experts, relationships with suppliers, trusted brands, etc. How much of the surplus will be captured by AI companies vs the owners of these complementary inputs? Optimistically: producers of general-purpose technologies often capture only a small fraction of surplus; and multiple frontier AI companies might sell similar products and bid each other down on cost.
Most of the economic surplus from AI occurs outside the US. The majority of these complementary inputs are situated outside the US. So most AI-driven economic value-add should occur outside US borders.
If (1) and (2) both hold, a significant fraction of AI’s economic surplus will accrue to non-US actors.
But how can middle powers guarantee frontier AI access? It’s tough, but a few strategies:
Build data centres. Partner with frontier AI companies to build secure data centres domestically, in return for guaranteed frontier access. This is a big win-win. AI companies improve their bargaining position with the US government. Recall, the US government threatened to destroy Anthropic when Anthropic insisted that their AI systems wouldn’t be used for legal mass surveillance.
Adopt AI. The more middle powers use frontier AI, the more costly it is for AI companies to cut them off.
Invest in frontier AI companies. Once they IPO, middle powers could invest billions or trillions into leading AI companies, in return for access guarantees.
Support the US internationally. If middle powers throw their diplomatic and military weight behind US foreign policy objectives, it benefits the US to keep them strong.
Build a relationship with China. If the US refuses to grant middle powers access to frontier AI, the national security implications are dire. Middle powers need a plan B, and China is the only other game in town for frontier AI. Only if this alternative is truly credible can it be leveraged into access to US frontier AI.
Ultimately, this involves middle powers threatening to sell semiconductor equipment and chips to China instead of the US. Obviously, that’s pretty far outside the Overton window. But that may change as the world rapidly wakes up to powerful AI and its national security implications.
Demand kill switches on US data centres. This is much more late-stage, after the world has truly woken up to the strategic implications of AGI. Suppose US and middle powers agree to a “chips for frontier access” deal – middle powers continue to supply the US with frontier chips; US continues to give middle powers access to frontier AI. The middle powers might still worry: what if the US suddenly changes its mind once it has superintelligence? By then, the US might be powerful enough to dominate without continued allied support. This is where kill switches can help. If the US withdraws AI access, allies could destroy US data centres in response. It’s a way to lock-in the deal.
(h/t AI futures project for this idea. A related idea is for US data centres to be placed in a location that’s easy to attack – like in space)
Beyond securing access to frontier AI, how else can middle powers maintain economic and military leverage?
Build physical infrastructure. Factories, robots, solar panels, batteries, semiconductors — all these industries are highly complementary to powerful AI.
Maintain nuclear 2nd strike capability. The point isn’t to use it. But it improves their leverage for stage 2.
The catch-all meta-point here is waking middle powers up to superintelligence.
I’m not recommending middle powers do their own frontier AI development. Seems very hard for them to catch up with the US.
Stage 2: Ensure that, when superintelligence is developed, it refuses to crush middle powers
If stage 1 goes well, middle powers remain somewhat powerful economically and militarily deep into the singularity. But it might fail. What can middle powers do if they see the US on track to total global dominance?
First, they should demand a pause/slowdown of AI development. But the US may refuse – pausing is very costly if alignment risk is low. And pausing is a stopgap: eventually, superintelligence will be developed.
An additional demand: when superintelligence is developed, it’s designed to refuse to crush middle powers. By doing this, the US would credibly bind itself to maintaining the sovereignty of other nations.
The optimistic case here is that this isn’t a big sacrifice for the US. They can still become as rich as they like and achieve their security interests. Sure, they can’t seize control of other nations, but that is not an important goal of theirs anyway. Losing that option is well worth the benefits: other nations cooperate economically, don’t attack US data centres, and don’t threaten nuclear war.
The pessimistic case is that this involves an insane degree of irrevocable hand-off to AI. The US must literally be unable to attack middle powers no matter how hard it tries: retraining the AI, turning it off, training a new more powerful AI, passing new laws, using the military to destroy the datacentres the AI is running on. For it to be truly binding, the US must permanently hand over military and political power to AI. That might be deeply unpopular, and indeed seem insane to the US. It’s also very hard to verify: you can’t just verify the training run, you need to verify that humans+other AIs have no way to disempower the trained AI. It’s more like verifying “who would win this civil war” than “technical property XYZ holds”.
The realistic path here probably involves gradually handing off more and more control to AI that refuses to crush middle powers, with no clear point at which humans could no longer wrest back control.
The longer middle powers wait to push for stage 2, the less leverage they will have because the US will have pulled further ahead economically and militarily. So they should be pushing in this direction constantly. Eg demanding transparency into the model specs of powerful AIs deployed in the US government, and arguing that powerful military AI should be designed to obey international law.
(I described the plan as involving two stages because that’s how I expect it to play out over time. But succeeding at either stage is sufficient! If middle powers stay economically/militarily competitive, they never need to bind US superintelligence. And if they do bind superintelligence, they won’t be crushed no matter how far behind they fall.
Is it good to avoid middle powers getting trounced?
I live in the UK, so I am biased here. I do not want the UK to become a supplicant to the US!
But here’s a brainstorm of pros and cons from a more impartial perspective.
Pros to empowering middle powers:
Avoid a single point of failure. If the US becomes globally dominant and its political system fails, that’s a global failure.
More democracies. Many middle power democracies look more robust than the US, so more middle powers may mean more democracy.
Improve the US. Middle powers will have an interest in maintaining free market democracy in the US. “Free market” bc they’ll want multiple AI companies competing to sell cheap API access to non-US countries. “Democracy” because they’ll expect that the US is more likely to maintain a strong alliance with middle power democracies if it stays democratic.
Experimentation. Experimenting with multiple different political and legal systems seems generally good for figuring out a good way to govern society post AGI.
Pause AI. They could potentially pressure US/China to pause/slowdown reckless AI development.
Prosocial norms. When multiple actors bargain with each other (e.g. about how to distribute space resources, whether to develop a dangerous technology), they tend to frame arguments in terms of pro-social norms, and so agreements tend to emphasise the actor’s more virtuous/ethical values.
Cons of empowering middle powers. Multipolarity has its own downsides:
more likely to lead to war.
can drive extreme competition, eg racing to develop a dangerous technology, or to hand off power to misaligned AI
Harder to prevent harms from offense-dominant technologies like bioweapons
my plan involves waking up middle powers, which could shorten timelines
I think that “middle power” is a bit too broad and too narrow for the question you’re asking. For example, on the “too broad” front:
There’s Iran, which is friendly with Russia and China, and certainly gets military support from them, but also infamously tried to compete with Russia for control over Syria in a way that ultimately compromised Syria’s ability to remain stable. It’s sovereign, culturally distinct from, and can occasionally pick outright fights with its allies, but it’s not in the same weight class as them.
There’s France, which has its own military, its own (non-frontier) LLM, and domestic industry, but America can (infamously, around five years ago) straight up veto their arms export deals when they become inconvenient, and, whether we like it or not, American culture is pretty dominant in France. French anti-US sentiment is essentially indistinct from the complaints of US liberals, for example.
There’s India, which doesn’t really have one patron, and punches below its weight militarily[1], but does have a large economy. Their relationship with China is adversarial, and their reputation in America is declining quickly, so their advantages/disadvantages are very different from those of the two countries above.
If an intelligence explosion is defined as a model’s capabilities increasing vertically, faster than humans can react, then it’s very likely that America/China are both much too large that even treating them as unified beneficiaries is plausible. If the model has its own “wants”, then the model is now in charge. If the model has been successfully aligned to answer to some group of people, then that group is now in charge. The window in which a model is sufficiently broad in its allegiance that it believes it is loyal to “all Americans”[2], but sufficiently narrow that it doesn’t care about anyone else, seems extremely small, and I don’t think that anyone—even the most fervent American patriot—is aiming for it.
If you define intelligence explosion more broadly, as your post appears to indicate, then I’d argue that “middle powers” get on as they’ve been getting on already. The benefits of the Russo-Chinese AI-industrial complex flow to Iran because they want Iran to survive, and Iran has leverage because these countries, again, want Iran to survive in its strategic position. Likewise, negotiations of terms aside, the EU essentially pushes in the same direction as the US on most if not all international matters, and while its domestic industry would be hollowed out significantly by a hypothetical leap in AI performance that America declined to share out, the products of American industry would definitely continue to be exported.
Seeing lackluster results in a recent conflict with Pakistan, which is six times less populous and doesn’t have nearly the same number of Western/Eastern arms suppliers
How do you define that, anyways? Is a guy who showed up on a boat one day ago American, even if he doesn’t care about America, doesn’t speak the language, and has no generational ties? If that’s the case, then you can just reclassify the entire planet as citizens and the issue’s solved.
I think this would make a good ‘top level post’, as they say
Thanks for the push—published!
If ASI is developed, they have a decent chance of avoiding this fate due to extinction from ASI misalignment or misuse, which imo is a much worse fate.
if we wanted to and kept wanting to, we could just ban AGI for a very long time and maybe forever. [1] instead of making non-human-descended top thinkers, we could just continue carefully becoming more capable and intelligent as humans. i don’t think this would be that weird. regulation of development (and specifically the development of intelligence) much more stringent than a very long AGI ban is probably going to happen in any future with intelligent life, because probably many aspects of
that said, i agree it is unlikely that we ban AGI in practice (i think it’s likely we make an AGI this century and that’s the worst thing ever done). i’m in part writing the present comment because a nontrivial contributor to it being unlikely that we get an AGI ban is that people think and say this sort of stuff. i’ve been irked by the same sort of fatalism about AGI in a bunch of other writing by forethought, 80000 hours, and many others in EA / AI safety and beyond. to clarify: i’m certainly not asking anyone who thinks an AGI ban is unlikely to lie about that, and i’m also not asking anyone to stop saying “an AGI ban is unlikely” when that is pertinent. but i think many people are weirdly systematically speaking as if an AGI ban is not an option worth considering, including many who in fact believe that it would be a good idea to ban AGI for a long time (if it could be done) or at least have significant probability on that
i think this is also a better thing for middle powers to fight for than what you propose (which i’d semi-seriously call a collaborationism-maxing policy package), for the sake of the world and also for their own sakes
ok maybe a landian/marxist/[nature red in tooth and claw] scenario where everything that isn’t competitive keeps getting annihilated is also possible, idk. tbh i just really wanted to state this antithesis to “AGI is inevitable”. maybe some qualifier like “if the future is not absolutely ruthless” needs to be added.
I have a hard time piecing together the context and the target of this plan. Is this plan designed to be convincing to certain people, vs. what are your beliefs about the world that underly it? (I don’t know who you are; if your views are somewhat known around here, sorry about that, I missed it.) You talk at the same time about superintelligence taken seriously, but then say that it will be built, and suggest (maybe, not clear on this point) that countries help it being developed. This looks plainly insane to me; but maybe you mean that countries won’t take superintelligence seriously, or are run by cursed madmen, so the plan you design is a complex plan that takes into account their wants and beliefs to feed them something shovel-ready that increases the likelihood they’ll do something better on your own terms, or are you also a madman that will not just stop pursuing superintelligence? I mostly agree with what I guess is the MIRI view here, that if I am another country and I believe in upper-case-S superintelligence, then if the US wants to go and build it what I do is start a nuclear weapon program to make it clear what the payoffs are.
I feel quite nervous about this plan (primarily “middle powers threatening to sell semiconductor equipment and chips to China instead of the US”).
Navigating the intelligence explosion feels hard enough as is. I’m worried that encouraging middle powers to play both sides will just lead to chaos.
I see where you’re coming from. The US is much less trustworthy than it used to be. I can imagine worlds where this plan would lead to good outcomes. However, my intuition is that the chances of this plan blowing up in our faces is way too high. Maybe, I’d change my view if I spent more time reflecting and digging into the facts, but this is where I am at the moment.
”For it to be truly binding, the US must permanently hand over military and political power to AI.”
Also, this is not something we want to rush. I struggle to see how this could happen fast enough to reassure the middle powers, whilst also being late enough that this wouldn’t be utterly reckless. The best counter I was able to come up with is that being able to counter threats might force a handoff anyway and that there might counterintuitively be no difference in the amount of trust you need for a temporary handoff vs. a permanent handoff given the ability of an AI to transform the former into the later.
If we’re in a sim, it’s being used for acausal trade
Me: Our world is exactly the kind of thing you’d simulate if you were doing acausal trade! It’s just before civilisation develops the ability to lock-in deals.
Sceptic: Sure, but there’s other reasons ppl might simulate earth. Maybe it’s for ppl’s entertainment? Maybe it’s social science, exploring alternate histories?
Me: For sure. But whatever the purpose of the sim is, it will contain info that’s relevant to ppl that want to do acausal trades. It will have info about who has power post-AGI, what their values are, and whether they want to do acausal trade. If someone ran the sim for entertainment, they’d obviously sell that info to the acausal trade folks.
Sceptic: Won’t the acausal trade folks just run their own sims?
Me: Maybe! But they’ll be keen to buy relevant info from others who runs sims. If others run earth sims for entertainment, the acausal trade folks will buy the info and run fewer earth sims themselves.
At this point we can’t be very sure that acausal trade is even a thing, since we don’t have a formal solution to decision theory we can be confident in (or even meet the lower bar of not having clearly serious flaws), and therefore can’t derive acausal trade from it as a solution to some decision problem. (Or have some other way to gain high justified confidence that acausal trade is actually a thing.)
And then even if acausal trade is a thing, whoever is running sims of us may not be running a decision theory that allows or recommends acausal trade, or we may not have anything to trade that they want, or the trade fails for some other reason like high “transaction costs”.
Do we need to be “very sure”? Seems like the OP doesn’t assert any enormous confidence.
I was reacting to the lack of hedging or explicit uncertainty in the first line “If we’re in a sim, it’s being used for acausal trade”. Possibly Tom didn’t mean to express very high confidence by it (I see that in a more formal occasion he does explicitly state his uncertainty about acausal trade[1]), but I feel an obligation to err on the side of caution here, in case some readers do interpret it as expressing high confidence.
Quote from the link: Of course, unlike the causal case, whether consensus goods get funded depends on whether agents want to do acausal cooperation at all—which depends on their decision theories and their beliefs about their degree of correlation with others.
Ah, cool, yeah, fair enough. I interpreted that as more of a title, and am used to titles usually skipping epistemic qualifiers because you really want them to be short.
Thanks, i’m not personally “very sure” either
But I wouldn’t rule out someone who’s thought a lot about DT being pretty confident. I don’t think you need to need “solve” DT to be v confident that acausal trade is a thing anymore than you need to solve ethics to know that murder is wrong.
I could imagine that some of the acausal trade crowd have thought long enough about the space of decision theories and their implications to conclude that acausal trade is a consequence of many plausible DTs and is very likely happening.
My understanding is that even with CDT you can get sim-based trade (which i’d consider a form of acausal trade), and that on a first pass EDT and UDT both imply that acausal trade makes sense. So we only need some powerful agents to do one of these decision theories for acausal trade to go ahead.
I guess I can imagine a countercase like “bc of threats very few civs do acausal trade”, though it’s hard to see it go down to zero. I’d be curious if you have other counter-cases in mind.
(In general i’d defer to someone who thinks about this more on how likely acausal trade is to happen overall!)
I’m not aware of anyone who has claimed this, and wouldn’t trust such claims anyway, since humans just aren’t that good at fully delineating the space of plausible solutions to some philosophical question. E.g. for the first half century of decision theory as a field, nobody thought that updatelessness might be worth investigating.
I wouldn’t defer a lot, because humans are also bad at finding flaws in their favorite philosophical ideas. Think of, e.g., theorists/proponents of Objectivism or Communism.
There are too many civilizations in the multiverse to simulate all of them. We can only do a sampling and then decide how to trade based on the statistical properties, but this gives an incentive to free-ride (i.e. getting the benefits of trade from other civilizations that did not specifically simulate us, without paying the costs), which may cause an overall breakdown in trade.
Hmm, but if each civ simulates 1 million others, every civ should be simulated 1 million times? I.e. the more civs in the universe, the more civs to do the simulating. And then i think there’s a stable equ where you only give nice things to the civs you simulate if you see them giving nice things to the ones they simulate. I suppose there’s a q of whether we’re able to achieve that equ?
Also, you can do the thing where when your sims run their own sims, you insert your universe into their sim, guaranteeing 2-way trade?
tiny nit: non-trade sims could happen late in the universe when the available lightcode isn’t big enough to impact trade much, or people could have already locked in their acausal trade decisions, or something analogous in another universe.
Interesting! Continuing this chain of thought…
But then, entertainment sims might oversample entertaining outcomes (e.g. rerun an event 10 times and use the branch with the most entertaining result). And then, as a result, acausal trade folks’ marginal willingness to pay for info about how those entertaining worlds turn out would go down. And for similar reasons, the stakes of our actions would then generally be lower in any given entertainment sim.
How do you “buy” info from another universe? They can’t respond.
It’s not from another universe, just from your neighbours who run different simulations.
Weak argument? The set of times that people incidentally produce information relevant to other people is much broader than the set of times they sell it to them.
I think I mostly agree with what this is actually saying, but I’m not sure it’s definitively acausal trade. e.g. it might be for just mundane trade, dividing gains by simulating Shapley values or something as part of a (very elaborate) cooperative bargain.
Forethought is hiring!
You can see our research here.
You can read about what it’s like to work with us here.
We’re currently hiring researchers, and I’d love LW readers to apply.
If you like writing and reading LessWrong, I think you might also enjoy working at Forethought.
I joined Forethought a year ago, and it’s been pretty transformative for my research. I get lots of feedback on my research and great collaboration opportunities.
The median views of our staff are often different from the median views of LW. E.g. we probably have a lower probability on AI takeover (though I’m still >10% on that). That’s part of the reason i’m excited for LW readers to apply. I think a great way to make intellectual progress is via debate. So we want to hire ppl who strongly disagree with us, and have their own perspectives on what’s going on in AI.
We’ve also got a referral bounty of £10,000 for counterfactual recommendations for successful Senior Research Fellow hires, and £5,000 for Research Fellows.
The deadline for applications is Sunday 2nd November. Happy to answer questions!
Accelerating tech progress + slow military procurement → cheap decisive strategic advantage
A common prediction of an intelligence explosion is that tech progress gets faster and faster.
When I speak to ppl from DC, I’m told that the government and military will be very slow to adopt new tech.
If these two things are both true, there’s a scary implication.
If tech progress speeds up by 30x relative to recent history, then a 3-year procurement delay by the military means they’re deploying tech that’s effectively 100 years outdated. Even a 1-year delay at that pace means your military is fielding equipment from a completely different technological era. The US military spends ~$1T/year. But with tech that’s 30–100 years more advanced, you could potentially defeat them at 1/100th of the spending — just $10B!
A rogue actor — a private company or a government clique bypassing standard procurement — could spend a tiny fraction of the official military budget on cutting-edge tech and potentially overmatch the entire conventional military.
There’s a massive untapped potential for cheap military dominance that no legitimate actor will exploit bc only the military has legitimate authority to procure weapons and they are bureaucratic.
Here’s a visualisation of the basic dynamic:
Possible solutions:
Military procurement needs to get dramatically faster during an intelligence explosion. New AI systems and AI-produced technologies need to be rapidly integrated into official military capabilities. This probably means automating the procurement process itself. This is counterintuitive from some AI safety perspectives — many people’s instinct is to delay military AI deployment.
Delay rapid tech progress until military procurement has become super fast + safe. This might in practice involve delaying the intelligence explosion and/or the industrial explosion as well
Better monitoring/surveillance to make sure no one is secretly building military tech
Others?
If everyone in our universe doing acausal trade coordinates, we can sell “cosmic real estate” for monopoly prices
Let’s assume that there are many different universes (or Everett branches) that acausally trade.
Some traders won’t about “resources in our civ’s future lightcone” linearly. As a toy example, the leader of a distant alien civilisation might want to get a statue of themselves in as many different other universes as possible.
If many different actors in our universe do acausal trade, and compete with each other to trade with the alien leader, then they’d bid down the price of building that statue. Whereas if they all band together, they could hold out for a much higher price. So it could be in our collective interests to coordinate and “price fix”.
This is an example of a civilisation-wide public good that could be important long into the future.
i notice a lot of disagree votes here—would appreciate an explanation as to why