Would concentration of power be bad, given offense-dominant weapons?
Title is a question because I’m looking for an answer, not providing it.
I wanted to stress-test some ideas and reason through some questions I have about concentration of power. These have been bubbling in the back of my head for a while, so I’ve taken two hours this morning to write some of them up and see how others feel.
The main project I’m currently working involves the mitigation of AI proliferation, particularly for offensive capabilities. The core issue is that a) advanced AI can plausibly allow you to make cheap, ultra-destructive weapons, and b) that the cost to train an advanced AI system declines quickly over time because of algorithmic efficiency improvements. As a result, the more you allow proliferation early on, the more invasive and ineffective enforcement becomes, until we hit a point where a few people are empowered to destroy the world at the expense of everyone else (maybe they built a fail-deadly, they’re terrorists, irrational, try to build a superintelligence and end up misaligning it, etc).
In my model of things, these destructive technologies are offense dominant (impluasibly inefficient or impossible to defend against), which means that the only realistic way to control them is to limit their spread preemptively. While it’s hard to predict the future, I’d expect that the first ASI labs to end up nationalized by the USG and PRC, meaning that offensive capabilities are initially monopolized. If they recognize the security interest of restricting AI proliferation (as the U.S and Soviet Union mutually agreed to do with nukes), they will use their strategic position to try and disempower other states/actors from getting access to offensive AI capabilities before they can become cheap to access. Perhaps they commandeer the entire AI supply chain, and put restrictions in place such that all future AI systems are aligned to the US-China coalition goals.
As far as I can see, this would reasonably secure the world against the threat of black ball AI tech. But an important secondary issue it creates is concentration of power, wherein the rest of humanity is subjected to the whims of the USG and PRC, because they have a monopoly on violence through their control of ASI technology and the commandeering they just did, and have to continue to do to in order to make sure an unauthorized ASI is never built.
Here are some of my intuitions about this situation:
First, there doesn’t seem to be a reasonable alternative to this sort of monopolization. Offense-dominant technology is only really curtailable by MAD style policy, which breaks down as the number of actors increases. If you want humanity to enjoy the benefits of powerful AI systems, then you need to make sure that the dual-use systems are preemptively controlled before they’re distributed, which means that they’re passed through some kind of central authority first. That central authority needs to have a strategic advantage over everyone else, in order to make sure that they’re genuinely not building the dangerous technology (think IAEA, sanctions, bombing Iranian enrichment facilities, etc).
As a simple example, consider allowing other countries/companies/individuals to train their own ASI. If this ASI might end up misaligned, and there are cheap weapons it can develop that can threaten everyone else, each of these actors is a danger to the whole of civilization. The only way to avoid this scenario is to disempower those future actors ahead of time, such that they can never build the misaligned ASI or comparable weapons in the first place.
Basically, I think d/acc (at least the defensive and decentralized components) is an unviable strategy, because you won’t be able to reliably defend against the widely distributed and cheap offensive technology ASI would allow. While I’m sure there are comprehensive, defense dominant solutions that can be widely distributed in some domains (cryptography being the classic example), I don’t think this is true of all future offensive technologies, which it needs to be. If there’s even a single cheap, existentially powerful technology that exists (and we already have promising candidates like mirror life or misaligned ASIs), then the plan of allowing AI to proliferate for defense will have the opposite effect of favoring bad actors.[1]
This doesn’t mean that ordinary people can’t be allowed to use any AI technology. They can! It just needs to be centrally processed and controlled first to make sure the dual-use stuff is restricted in some way (certain robotics, AI r&d, biology, engineering, and material sciences applications most likely). This looks something like allowing the use of superhuman medical researchers to create wonderdrugs, although the program is backdoored/pre-aligned/monitored to make sure it cannot comply with requests to develop bioweapons.
Second, it seems to me that most people might end up ok despite the concentration of power required by having governments monopolize ASI, even in the most extreme dicatorial scenarios.
The primary reason is that, given the abundance that ASI might enable, both in the efficient production of goods and explosive supply of labor, it would become very cheap (as a fraction of GDP) to have most people live extraordinarily good lives. Eliezer has made the point before that misaligned ASIs would have no reason to spare even a tiny fraction of their resources, because they won’t have even a tiny preference to satisfy. While I think this is likely true of misaligned ASIs, it seems wrong to argue that the same is true of most humans, who do have some nonzero preference for being altruistic along with their other goals. If we end up in a situation in which almost all labor ends up controlled by a handful of company executives/politicians, it would only take a few of them committing to a permanent welfare payment for everyone to live great lives (at a small fraction of their actual resources equivalent to their preferences).
A common criticism of this altruist/redistributionist perspective is that Moloch devours all, and that those who don’t advocate for human welfare will eventually outcompete those that do. However, it seems unclear that there’s a realistic mechanism for the non-welfare advocates to do this to the altruist decision makers, given the presence of offense-dominant weapons at the limit of intelligence. Even though the U.S has unbelievable conventional military superiority to North Korea, for instance, the fact that they have nuclear weapons means that we cannot arbitrarily impose our preferences about how North Korea should act onto them. Although the U.S might “win” a nuclear exchange with North Korea, the damage it would be able to cause to the things we value in the process is so large that we don’t take the steps that would disempower them in the first place.
Roughly speaking, you can swap out “U.S” and “North Korea” with “Optimizers” and “Altruists”. Although there might be some (or even most) decision makers who would prefer to get rid of all the strategically/politically disempowered humans who are consuming resources to better optimize for more galaxies/inter elite status games/etc, there probably won’t be a decisive action they can take against civilians or the altruists that advocate for them that doesn’t end in catastrophic retaliation. If an altruist has their hands on offense-dominant ASI technology, then they can just commit to using it in retaliation if they confirm that the optimizers are trying to disempower them. As a somewhat modern parallel, it’s reasonable for the U.S to discard plans to assassinate Kim Jong Un on the basis that it wouldn’t be able to stop a retaliatory strike when the regime panics. And given the coordination/verification mechanisms of superintelligences for upholding these contracts, rather than relying on human decision making in the moment, you could probably get pretty ironclad guarantees about the behavior of other actors in spite of their preferences.[2]
This situation would still lead to unimaginable imbalances in material wealth, probably many orders of magnitude more extreme than today. But I don’t believe that inequality is intrinsically problematic from a welfare perspective: it’s far more important that the people at the bottom meet the absolute threshold for comfort than it is for a society’s Gini coeficient to be lower. The fact that Elon Musk can afford to send his kids to university for millions of years doesn’t detract from the four years I spent attending on scholarship. The only direct threat the ultra-wealthy pose to my wellbeing is their capture of the state/important institutions that I use, which in the above scenario is compensated for by altruists who can cheaply afford for me to be happy and are able to threaten something their competitors care about over it.
Given all of this, concentration of strategic power seems both necessary and… probably ok? Even in this worst-case scenario, where power is totally monopolized by a handful of people (say the Oversight Committee from AI 2027), all it takes is a small preference among some of them for civilians to live great lives for them to be able to do so. The state needs to clamp down on offense-dominant offensive capabilities to make sure that non-state actors never get access to them (or else someone will eventually use them), but afterwards, beneficial AI technology can be allowed to mostly flourish.
Of course, I still think it would be better to have outcomes like value-lock-in of welfare/political rights for all humans at the inception of ASI, or for the U.S and China to somehow avoid intent alignment and instead aim straight at civilizational alignment where the ASI runs everything itself, rather than having to account for the preferences of very few. But I believe the above situation—nationalization of labs, intent aligned AI, realization of the strategic importance of non-proliferation, permanent monopolization of strategic power—represents the default path to dealing with the proliferation problem, so it seems important to reason about what it might look like and whether that’s acceptable.
- ^
As an example of the kinds of defensive plans I’m talking about, I’m referring to Luke Drago and Rudolf L’s proposal for defensive augmentation, or Helen Toner’s adaptation buffers, where we prepare society ahead of time through the wide-scale distribution of technology.
I have several thoughts on this, which I’ll explore in a future, properly edited post on the strategic implications of proliferating powerful AI systems. My main critique is that I do not share their optimism that you can efficiently respond to all existential threats, given the cost-imbalance of defending against them. As the number of threats and actors increase, spending on defense has to grow exponentially. Offensive investments, on the other hand, are linear, since they can focus all of their effort on targeting the areas that are under-invested or even impossible to defend against. Ie, it’s much cheaper to try and overwhelm defensive capacity by just launching two pandemics at the same time than it is to make sure you are prepared for multiple pandemics.Analogously, we can think about how ridiculously inefficient trying to intercept nuclear ICMBs is, and how you could obsolete the hundreds of billions of dollars worth of interceptor research you’d need by a) building faster missiles, b) using more missles, or c), just putting your bomb in a shipping container and docking it next to Washington.
And even if you secure one avenue (wastewater monitoring, DNA synthesis screening, and KYC ends up being enough defense-in-depth for bioweapons that it’s basically solved), there’s no guarantee you can solve them all. If you upgrade the problem from a pandemic to mirror life, for example, outbreak monitoring and antibiotic response plans start to break down completely. And so on for swarms of insect-sized LAWs, nanotechnology, misaligned ASIs, and the unknown-unknowns of weapons development that lurk at the top of the tech tree. The only reasonable solution to this problem is just to make sure other people don’t have weapons in the first place, or preemptive non-proliferation.
I do not have the confidence to distribute general ASI without a guarantee that people will not have access to these offense-dominant technologies, and the only way to make sure that their AI cannot do those things would be for a more powerful central authority to have achieved ASI first and used their first-mover advantage to strip all future offensive capabilities away from other actors.
- ^
Similarly, even if the Chinese government decided to try to be more competitive by ignoring the welfare of its own citizens, it wouldn’t necessarily be able to impose its will on the U.S given the disproportionate leverage of offense-dominant weapons.
Hi Felix! I’ve been thinking about the same topics for awhile, and came to pretty much the opposite conclusions.
No nononono. So many people making this argument and it’s so wrong to me.
The thing is: altruistic urges aren’t the only “nonzero urges” that people have. People also have an urge to power, an urge to lord it over others. And for a lot of people it’s much stronger than the altruistic urge. So a world where most people are at the whim of “nonzero urges” of a handful of superpowerful people will be a world of power abuse, with maybe a little altruism here and there. And if you think people will have exit rights from the whims of the powerful, unfortunately history shows that it won’t necessarily be so.
I think we’ll never be at a point where a handful of people can defeat the strongest entities. Bioweapons are slow; drone swarms can be stopped by other drone swarms. I can’t imagine any weapon at all that would allow a terrorist cell to defeat an army of equal tech level. Well, maybe if you have a nanotech-ASI in a test tube, but we’re dead before then.
It is however possible that a handful of people can harm the strongest entities. And that state of affairs is desirable. When the powerful could exploit the masses with impunity in the past, they did so. But when firearms got invented, and a peasant could learn to shoot a knight dead, the masses became politically relevant. That’s basically why we have democracy now: the political power of the masses comes from their threat-value. (Not economic value! The masses were always economically valuable to the powerful. Without threat-value, that just leads to exploitation. You can be mining for diamonds and still be a slave.) So the only way the masses can avoid a world of total subjugation to the powerful in the future is by keeping threat-value. And for that, cheap offense-dominant weapons are a good thing.
Making an analogy with altruism here is strange. North Korea is a horrifying oppressive regime. The fact that they can use the nuke threat to protect themselves, and their citizens have no analogous “gun” to hold to the head of their own government, is a perfect example of the power abuse that I described above. A world with big actors holding all threat-power will be a world of NKs.
There’s a standard response to this argument: namely, inequality of money always tries to convert itself into inequality of power, through lobbying and media ownership and the like. Those at the bottom may have comfort, but that comfort will be short lived if they don’t have the power to ensure it. The “Gini coefficient of power” is the most important variable.
So yeah, to me these all converge on a pretty clear answer to your question. Concentration of power, specifically of threat-power, offense-power, would be very bad. Spreading it out would be good. That’s how the world looks to me.
It sounds like a big crux for this might be whether weapons would be offense-dominant enough to threaten the whole of society.
I basically think that this is very unlikely to be the case. It’s just more cost effective to whip up offensive weapons than it is to reactively defend against them. In the basic case of a bioweapon, someone can covertly develop several pathogens at once, which each need to be responded to with a specific vaccine/containment measures. All the onus is on the defender to predict the attacker, while the attacker can take their time and pool their resources in places that are currently underdefended. The attacker also only needs to succeed once, while the defender needs to succeed continuously.
With some advanced technologies, like mirror life or an unaligned ASI, there might simply not be any realistic reactive response. Just as it’s functionally impossible to stop another country from nuking you today if they really want to, I strongly suspect that future offensive technologies will put you in a MAD style equilibrium. It’s just that the existentially threatening technologies that underly MAD will be much cheaper to access in a world full of superhuman AI assistance.
It seems pretty plausible to me that a handful of people or a terrorist organization really could end the world if they put their minds to it, given they have access to general superintelligence. The easiest way might just be developing a misaligned ASI intentionally, and have it run around causing maximum difficulty for the aligned ASIs that are paying the alignment tax.
I think that the masses have always had a substantial amount of threat value, and that democracy won out not because they became more threatening but because their economic value increased so greatly during the industrial revolution. Countries like China developed gunpowder all the way back in the 9th century, but they remained a monarchy for a thousand years afterward. In fact, the country saw multiple successful peasant revolutions throughout its history (including the establishment of the Ming Dynasty) but no establishment of a democratic government, because there were no economic incentivies for the new leaders to overhaul the system. It seems to me that there might be ways to preserve democracy that are not conditioned on the distrubtion of literal strategic power.
Can we really say that the U.S is democratic today primarily because firearms are so common? If that were the case, why have its cultural neighbors like Canada and European countries arguably been more democratically stable? The west is democratic because economics rewarded it, and then that incentive lead to a culture that valued democracy more or less for its own sake.
This is a good point, and one I underconsidered. If you have a world where a handful of people go around allocating resources according to their preferences, you will end up with pockets of altruism and oppression relative to how much preference for these the power players have. I don’t think I put enough weight on some of the players wanting actively bad outcomes.
I will say that NK is itself an example of a group that would be far better off if it had never been allowed to obtain its offense-dominant weapons. In a counterfactual world where the U.S was the only state with nukes and had succeeded at ironclad nonproliferation, you would probably see a proper pax atomica. Your take seems to be that the solution to the existence of potential states like NK is to diffuse strategic power further so that individuals are empowered, but I think it provides the opposite lesson: that we should learn from the mistakes we made with nukes and actually constrain the spread of existentially powerful technologies as much as possible this time.
Allowing everyone unfettered access to ASI is pretty close to giving each citizen in North Korea a nuke: sure, the tyranical government wouldn’t be effective, but now everyone is at the mercy of people with evil or irrational values.
--
Other assorted points:
Thanks for taking the time to write this stuff up! I try to spend time thinking through what’s going to be valuable/what we can afford to trade off on, and it’s good to get reality checked.
I don’t personally hope or want the scenario in the essay to happen: it’s more of a thought experiment about where truly extreme concentration of power would be maximally bad. I think that you would essentially avoid the worst outcomes of gradual disempowerment in the scenario I listed (the part where almost everyone’s resources are below subsistence and starve), because it could be kept in check ~permanently by a commitment towards MAD from a wealthy philanthropist, but this clearly isn’t an ideal way to distribute resources from a welfare perspective.
There’s probably something of a middle ground we both agree with here: something like Seb Krier’s Bargaining at Scale, where most people are empowered with individual AI representatives and the government intervenes only to enforce contracts and preserve basic rights. I think that this sort of solution would mostly eliminate the black ball problem I was focused on, while still preserving most of the upside from tech diffusion.