That is the hope that the US and China, out of similar self-interest, agree that only they will be allowed to develop dangerous AGI (or in your framing, to restrict development of AI that can develop dangerous weapons),. This currently seems unlikely, but if the dangers of proliferation are really as severe as I fear, it seems possible they’ll be recognized in time, and those governments will take rational actions to prevent proliferation while preserving their own abilities to pursue AGI. The uneasy alliance between the US and China might be possible because there isn’t really a lot of animosity; neither of us really hates the other (I hope—at least the US seems wary of china but not to really hate it, despite it being fairly totalitarian).
There are many reasons that I believe that U.S-China cooperation on nonproliferation is likely, not all of which made it into the post. Specifically:
1. Great power agreements on nonproliferation are how we’ve handled all the most powerful dual-use technologies in the past. The U.S and Soviet Union could both recognize that the spread of nukes (and later, bioweapons) would be existentially dangerous to them both, and were able to work together to restrict them. And this was despite the fact that the Soviets were much more ideologically and economically distant than the U.S and China are today.
2. The relationship between China and the U.S has been tested before and not broken. Even when the public and most of his own party turned against Bush in 1989 over the response to Tiananmen, the executive branch was able to stay steady enough to continue to secretly work with top Chinese officials (to prevent them from turning to the Soviet Union for allies). Even if animosity between the countries reaches a similar boiling point again, there’s probably still room to handle natsec-critical talks privately.
3. The last concern, and where I’m the most uncertain, is on the question of speed: do officials in both countries recognize what’s going on fast enough to start doing nonproliferation work? On the one hand, the gov might get completely blindsided by the speed of progress, or perhaps become entirely politically captured by Nvidia’s lobbying. On the other hand, we haven’t seen any strategically relevant capabilities actually demonstrated yet: once they are, it seems difficult for the natsec apparatus to justify leaving them in private hands and not restricting who has access to them. Any steps the state takes to assert its domestic monopoly of violence will naturally dovetail into trying to do the same overseas.
If the government gets involved at all, asserting control over dual-use AIs seems like the most primal, basic step they can take: if they don’t have control of the ASIs/superweapons, then they’re probably not in charge enough to do anything else.
Of course this leaves the good reasons for Fear of centralized power. But it may be the lesser of two dangers.
Building off that previous point: it seems very difficult to imagine the government doing anything without the monopolization of strategic power, at least at the domestic level. Throughout the entire process of assembling a nuclear weapon, for example, the government always maintains its monopoly on force—hence why we have policies in place like making sure that defense contractors for nukes are never allowed to finish assembling them in-house. The reasoning is obvious: if someone other than the state has nukes, how can the state enforce any rules on them?
In a similar defense-contractor setup with the frontier labs (such as in a national superintelligence project), the buck would have to stop with the government. If you want the state to do anything, the state needs to keep ultimate control of what the ASIs are being used for. If anyone other than the state has the final say on the ASI’s commands, then the state isn’t in charge anymore and there was no point in bringing the government on to regulate things to begin with.
Rather than solve centralization of power by having multiple ASI projects competing with each other, I think a lot more attention should be on how we can diffuse power of a single ASI initiative. With nukes, the government technically has the capacity to blackmail anyone it wants domestically: if enough people in the chain of command coordinated, they could hold the rest of society hostage. But by making that chain of command wide and complex enough, it becomes difficult to unilaterally use your monopoly on violence in harmful ways.[1] This gives you most of the upsides with fewer risks: the gov is powerful enough to enforce nuclear nonproliferation, but has difficulty using its strategic monopoly to extractively bargain with people.
When combining this problem (the fact that the government needs to maintain its monopoly on force to be an effective regulator) with the fact that future AI systems will enable all sorts of offense-dominant strategies (such that d/acc-style proposals to build defensive capacity fail because too many people have AIs capable of designing cheap WMDs), it seems like the focus should be on making the government more coup-proof, rather than increasing the number of intent-aligned superintelligences.[2]
My one nitpick is that the framing in this post seems to leave aside the possibility of general-purpose AI, that is, real AGI or ASI. That presents solutions as well as problems; it can be used to improve security dramatically, and to sabatoge other nations’ attempts at creating powerful AI in a variety of ways. This may add another factor that goes against proliferation as the default outcome, while adding risk of totalitarian takeover from whoever controls that intent aligned AGI.
For proliferation as a default outcome, I mean as an outcome without much government intervention. Prices get low, more people build/steal strong models. I agree that you could use an ASI to sabotage the AI projects of other countries, but it is certainly a huge amount of government intervention. This is a possible “permanent solution” I mentioned in the post though, and I’ll spend some time in future articles weighing out the merits of strategies like the front-runners sabotaging everyone else, taking over supply chains, etc to enforce nonproliferation.
Also worth noting that the government has a long cultural tradition and political experience with managing the use of powerful weapons. This might still not be robust enough for superintelligence, but it’s a much more likely bet than the AI labs’ internal political structures, which immediately collapsed the first time they were seriously tested.
The number of intent-aligned ASIs can be greater than 1 and the future can still be good: in fact, by default I expect that this number is at least two, one each in a U.S and Chinese project. You can probably increase this number quite a bit (such as by letting big blocs of states get their own, like how Britain and France were given nukes) and still have a functional equilibrium, but it probably can’t increase to the level of “everyone in the public” or even “everyone who can afford a $100 million training run.” At some point you need the states to step in and stop people from misusing powerful models, which they can’t do if you allowed them to proliferate beforehand. The only effective defenses against most superweapons will be the common-sense answer of not letting them be built in the first place.
There are many reasons that I believe that U.S-China cooperation on nonproliferation is likely, not all of which made it into the post. Specifically:
1. Great power agreements on nonproliferation are how we’ve handled all the most powerful dual-use technologies in the past. The U.S and Soviet Union could both recognize that the spread of nukes (and later, bioweapons) would be existentially dangerous to them both, and were able to work together to restrict them. And this was despite the fact that the Soviets were much more ideologically and economically distant than the U.S and China are today.
2. The relationship between China and the U.S has been tested before and not broken. Even when the public and most of his own party turned against Bush in 1989 over the response to Tiananmen, the executive branch was able to stay steady enough to continue to secretly work with top Chinese officials (to prevent them from turning to the Soviet Union for allies). Even if animosity between the countries reaches a similar boiling point again, there’s probably still room to handle natsec-critical talks privately.
3. The last concern, and where I’m the most uncertain, is on the question of speed: do officials in both countries recognize what’s going on fast enough to start doing nonproliferation work? On the one hand, the gov might get completely blindsided by the speed of progress, or perhaps become entirely politically captured by Nvidia’s lobbying. On the other hand, we haven’t seen any strategically relevant capabilities actually demonstrated yet: once they are, it seems difficult for the natsec apparatus to justify leaving them in private hands and not restricting who has access to them. Any steps the state takes to assert its domestic monopoly of violence will naturally dovetail into trying to do the same overseas.
If the government gets involved at all, asserting control over dual-use AIs seems like the most primal, basic step they can take: if they don’t have control of the ASIs/superweapons, then they’re probably not in charge enough to do anything else.
Building off that previous point: it seems very difficult to imagine the government doing anything without the monopolization of strategic power, at least at the domestic level. Throughout the entire process of assembling a nuclear weapon, for example, the government always maintains its monopoly on force—hence why we have policies in place like making sure that defense contractors for nukes are never allowed to finish assembling them in-house. The reasoning is obvious: if someone other than the state has nukes, how can the state enforce any rules on them?
In a similar defense-contractor setup with the frontier labs (such as in a national superintelligence project), the buck would have to stop with the government. If you want the state to do anything, the state needs to keep ultimate control of what the ASIs are being used for. If anyone other than the state has the final say on the ASI’s commands, then the state isn’t in charge anymore and there was no point in bringing the government on to regulate things to begin with.
Rather than solve centralization of power by having multiple ASI projects competing with each other, I think a lot more attention should be on how we can diffuse power of a single ASI initiative. With nukes, the government technically has the capacity to blackmail anyone it wants domestically: if enough people in the chain of command coordinated, they could hold the rest of society hostage. But by making that chain of command wide and complex enough, it becomes difficult to unilaterally use your monopoly on violence in harmful ways.[1] This gives you most of the upsides with fewer risks: the gov is powerful enough to enforce nuclear nonproliferation, but has difficulty using its strategic monopoly to extractively bargain with people.
When combining this problem (the fact that the government needs to maintain its monopoly on force to be an effective regulator) with the fact that future AI systems will enable all sorts of offense-dominant strategies (such that d/acc-style proposals to build defensive capacity fail because too many people have AIs capable of designing cheap WMDs), it seems like the focus should be on making the government more coup-proof, rather than increasing the number of intent-aligned superintelligences.[2]
For proliferation as a default outcome, I mean as an outcome without much government intervention. Prices get low, more people build/steal strong models. I agree that you could use an ASI to sabotage the AI projects of other countries, but it is certainly a huge amount of government intervention. This is a possible “permanent solution” I mentioned in the post though, and I’ll spend some time in future articles weighing out the merits of strategies like the front-runners sabotaging everyone else, taking over supply chains, etc to enforce nonproliferation.
Also worth noting that the government has a long cultural tradition and political experience with managing the use of powerful weapons. This might still not be robust enough for superintelligence, but it’s a much more likely bet than the AI labs’ internal political structures, which immediately collapsed the first time they were seriously tested.
The number of intent-aligned ASIs can be greater than 1 and the future can still be good: in fact, by default I expect that this number is at least two, one each in a U.S and Chinese project. You can probably increase this number quite a bit (such as by letting big blocs of states get their own, like how Britain and France were given nukes) and still have a functional equilibrium, but it probably can’t increase to the level of “everyone in the public” or even “everyone who can afford a $100 million training run.” At some point you need the states to step in and stop people from misusing powerful models, which they can’t do if you allowed them to proliferate beforehand. The only effective defenses against most superweapons will be the common-sense answer of not letting them be built in the first place.