Given the level of political will and international coordination in the story, why can’t they just dismantle the compute supply chain?
If I understand correctly, the main argument agains Plan S is that at some point the global pause agreement will break down, and then we will be back where we are right now, and the race restarts again at a break-neck speed.
But what if part of the pause deal is that, both in China and in US allies, we destroy a large chunk of the existing GPUs, destroy the fabs, destroy the cutting-edge EUV machines, destroy the equipment necessary to build the EUV machines and disperse the teams working at all these companies so institutional knowledge is lost?
Once the compute supply chain is dismantled, governments can pay attention that no new cutting-edge chip fabs or necessary equipments are made—something that seems much easier to enforce than the restrictions on dangerous algorithmic progress in Plan A.
My understanding is that this wouldn’t have huge effects outside the AI industry. While there would be a huge stock market crash and it would be expensive to compensate the the affected companies, I’m not sure the financial loss would be more than 2x bigger than the stock market crash coming from all the priced-in growth being stopped in Plan A.
With the compute supply chain dismantled, the pause deal falling apart looks much less worrying. I expect that even if everyone started to try going full-speed when the deal fails, it would still take 10-20 years to rebuild the compute supply chain and train cutting-edge AIs again. During those 10-20 years, the pause deal can be revived again, and generally it seems good to have this longer time.
So I think the biggest question here is whether we expect the world to generally get better or worse during a long pause. I expect that the world is probably going to get better: I find the long-term historical trends encouraging, and I think that if we try to build AGI a hundred years from now, that will likely go better than if we try now. (Especially if we do some genetic engineering in the meantime.)
If they have the political will to do Plan A, they very well might also have the political will to dismantle the compute supply chain. This would be a variant of Plan S.
I think this is plausibly as good or better than Plan A, not sure. One issue with it is that a covert project with, say, 100k GPUs doesn’t really confer much geostrategic advantage in Plan A, but in Plan S, it might. Imagine: It’s 2040. The economy has recovered from the compute supply chain being dismantled; people have learned to live without computers. But negotiations for how to restart AI progress safely and transparently and in a power-distributed way are dragging on and on and it seems like it’s basically never going to reach agreement. Meanwhile, the covert project has managed to make an OOM or two of algorithmic progress since 2030, and is just a few years away from fully automating AI R&D, at which point they’ll probably have ASI within a few years of that...
Idk. We have a model of AI progress + model of black sites that tries to model situations like this. I don’t think it’s obvious either way how it would go.
My other objection is that I feel a lot of despair when thinking in near-mode about the Plan A proposal of slowing down algorithmic progress.
----
India’s leading company publishes a paper about a training dataset they used to make the user-experience for their AI smoother. The Indian regulator apparently green-lighted it, but they are known for their lenient approach. Some academics notice that the Indian model is now not just smoother to use, but it subtly feels like it’s smarter than other models, even though it doesn’t directly show on the standard benchmarks. The American AISI starts getting emails from academics, explaining that they feel like the Indian model got smarter, and that they think the paper the Indians published was too high-level to fully reconstruct what they did.
Meanwhile the American AISI is also getting emails from academics complaining about how the European AI is too biased against minorities, and how the Brazilian AI has too high persuasive capabilities, and how the compute cluster in the ocean is harming the fish. They are also tasked with defending the US companies from the totally unfair complaints coming from China and India that they are not transparent enough about their research.
The American AISI only has 200 people at this point, because they started out small, and you can’t easily grow an organization more than 2x per year. Also, 50 out of the 200 people joined because they care about the ocean cluster harming the fish. (It is true though that probably it helps a bunch that they have good AI assistants. Maybe the crux is that I don’t believe it that much that the AI assistants will solve the dysfunctionality.)
The American AISI starts a bilateral conversation with the Indian regulators, but by the time they get to anywhere, it looks like the Europeans and two American companies have already probably copied something like what the Indians did. The world clearly didn’t end, and the AIs are just a bit smoother to use now. So no one escalates to a big diplomatic fight, and they quietly accept that the Indians didn’t publish enough details this time. Everyone keeps scaling up the technique with more compute and more data.
A month later, there is a new report which indicates that some interpretability techniques show that some scheming propensities are maybe going up, maybe linked to the new technique. After another month (do you know how slow it is to do anything in a government agency?) the American AISI calls up the President, and tells him to put pressure on American companies and other world leaders to roll back the new technique. Unfortunately, the President doesn’t have much time, because the Australian PM is pressuring him to make sure no one’s AI goes along with human rights violations in an ongoing war in the Middle East. Plus, he needs to speak at an environmentalist conference about the fish. (Also, he is still the President, running the non-AI aspects of the country.)
Now the President needs to call up the other world leaders and make it clear that he is willing to go to war if needed, because this interp scheming score really shouldn’t go up (unlike last time when a bias score was going up, but after a lot of yelling, people decided it’s not worth fighting over). So the President and other world leaders (reminder: Xi is 73, Modi is 75, Trump is 80, Biden left office at 82) get together and discuss the interp results and come to an agreement.
----
I think probably the sheer amount of bureaucracy still slows down algorithmic progress in the companies, and probably the government agencies pushing for safer directions and trying to ban more unsafe directions will have a non-zero correlation with actual safety. But it’s going to be a mess, and I don’t think you can slow down progress more than 2x this way, and the companies and countries that are more willing to skirt the rules will come out ahead more.
In contrast, GPUs, fabs and EUV machines are physical objects. I have much more trust in the government to competently control them and slow progress that way than through governing algorithmic progress.
I think I’m just not as afraid of covert projects. In the world that has enough political will for Plan A or compute supply dismantling, I think it’s likely that neither the US nor China even wants to attempt a covert project, because they are both genuinely afraid of AI. And I don’t see who else could attempt a covert project with 100k hidden GPUs.
And once the compute chain dismantling happens, it becomes very obvious that the leading powers are extremely serious about not wanting AI. I think it’s over 70% likely that espionage will uncover the existence of a covert project, and then they will very likely get bombed in the world where everyone was so serious that they already blew up their own EUV machines. I expect that the US and China will be aware of this and not attempt a covert project.
(It’s also my understanding that GPUs probably mostly break after about 5 years of use. That will really hurt the covert project if they can’t source new GPUs, no? I don’t remember this being addressed in the supplement, but maybe I missed something.)
Also, even if a covert project manages to build something smarter than anyone else has, that’s not necessarily catastrophic. If they manage to build such a strong superintelligence that it can build invincible nanobot-armies in a basement without any previous industrial build-out, that’s bad. But if they would just want to do the usual strategy of building robots to build better robots to build more compute and drone armies, that will very likely be visible an be noticed, in a world that otherwise doesn’t have advanced compute an robotics, and will lead to bombing.
My understanding is that in this way, Plan A is more vulnerable to covert projects than the compute dismantling plan: in Plan A, if someone secretly builds a smarter AI than anyone else has, they can use it to devise a plan to bring a few percent of the huge compute clusters outside the easily bombable places, and then hook up the secretly built smart AI to the compute and the already existing huge robot economy and drone army. The path to victory seems much harder in the world without a lot of compute and robots.
In the world that has enough political will for Plan A or compute supply dismantling, I think it’s likely that neither the US nor China even wants to attempt a covert project, because they are both genuinely afraid of AI.
The game theory doesn’t work out this way. In two player models, the US and China can always each get a geopolitical power advantage from developing an AI more powerful than socially optimal, and so have an incentive to defect.
If they believe AI is very dangerous, they will just set the treaty capability cap lower, and still defect on it.
Sure, but it’s worth noting that’s a different reason. IMO fear of AI sets the perceived socially optimal outcome, and honor, collusion, transparency, etc determine whether a treaty holds.
Your proposal would be net positive but the question is whether it’s better than Plan A. Even if takeoff is very slow post-deal if you didn’t advance ai capabilities much during the deal or use them heavily to do useful work then you get much less benefit from the deal than in Plan A.
I think it’s pretty likely that if you start a very serious pause, with compute chain dismantling, then the race doesn’t restart for a very long time. And in the meantime, the world might just become a better place by default. I find it relatively plausible that in the next hundred years, the world would make as much progress towards democracy, stability and peace as Europe made since 1926. If US and China and the other great powers of 2126 will be on as friendly terms as France and Germany are now, then I will feel much better about slowly restarting AI development. If you think that the deal is very likely to break early, or if you don’t believe that the default secular trends in the world point in a good direction, then admittedly this less promising.
I’m not sure the financial loss would be more than 2x bigger than the stock market crash coming from all the priced-in growth being stopped in Plan A
I think it should be if people were well-calibrated? (Which is more likely in worlds where people are taking things seriously enough for these proposals.)
Delaying ~infinity growth by 100 years seems more than twice as bad as delaying ~infinity growth for 10 years. With exponential discounting, the only way I can currently see for this to be false (though there’s probably other ways) is if you think that most of the value of “~infinity growth now” would already have been lost by moving to “~infinity growth in 10 years”. But that doesn’t seem consistent with how low real interest rates are. (They tend to value stuff in 10 years at more than half as much as stuff now.)
And plan A looks better than “singularity in 10 years” by any very time-sensitive financial measure, since you get lots of growth in the intermediate 10 years too.
Yes, that’s fair. But given that Plan A starts with a temporary full shutdown, and it will be generally hard to know what pace of progress will be allowed later, I think markets will probably initially think that there is substantial chance that the temporary pause won’t really be lifted or progress will be reversed in other ways. (E.g. I don’t expect markets to be enthusiastic about the proposal to put all data centers in Mongolia, under the threat of Chinese bombing.) So I think the initial stock market crash at the announcement of Plan A will still be huge—I would now guess between 30% and 50% as big as in the case of full dismantling of the compute supply chain.
I think one of the main political barriers to starting either the compute dismantling or Plan A is the unpopular initial crash and the corresponding industry outcry, and that will comparably be huge in both cases.
But given that Plan A starts with a temporary full shutdown, and it will be generally hard to know what pace of progress will be allowed later, I think markets will probably initially think that there is substantial chance that the temporary pause won’t really be lifted or progress will be reversed in other ways.
Sure. Though conversely, there should also be a big probability that the pause will be immimently lifted and progress will keep going just as before. Whereas there’s much less hope of that if there’s an announcement to destroy the compute supply chain.
I suppose that if the initial announcement comes with a 30% probability that all compute will be destroyed, then the crash should probably be at least 30% as large as a definite announcement that the compute will be destroyed? Though 30% on that given the initial pause seems maybe 2x too high to me or something.
Given the level of political will and international coordination in the story, why can’t they just dismantle the compute supply chain?
If I understand correctly, the main argument agains Plan S is that at some point the global pause agreement will break down, and then we will be back where we are right now, and the race restarts again at a break-neck speed.
But what if part of the pause deal is that, both in China and in US allies, we destroy a large chunk of the existing GPUs, destroy the fabs, destroy the cutting-edge EUV machines, destroy the equipment necessary to build the EUV machines and disperse the teams working at all these companies so institutional knowledge is lost?
Once the compute supply chain is dismantled, governments can pay attention that no new cutting-edge chip fabs or necessary equipments are made—something that seems much easier to enforce than the restrictions on dangerous algorithmic progress in Plan A.
My understanding is that this wouldn’t have huge effects outside the AI industry. While there would be a huge stock market crash and it would be expensive to compensate the the affected companies, I’m not sure the financial loss would be more than 2x bigger than the stock market crash coming from all the priced-in growth being stopped in Plan A.
With the compute supply chain dismantled, the pause deal falling apart looks much less worrying. I expect that even if everyone started to try going full-speed when the deal fails, it would still take 10-20 years to rebuild the compute supply chain and train cutting-edge AIs again. During those 10-20 years, the pause deal can be revived again, and generally it seems good to have this longer time.
So I think the biggest question here is whether we expect the world to generally get better or worse during a long pause. I expect that the world is probably going to get better: I find the long-term historical trends encouraging, and I think that if we try to build AGI a hundred years from now, that will likely go better than if we try now. (Especially if we do some genetic engineering in the meantime.)
If they have the political will to do Plan A, they very well might also have the political will to dismantle the compute supply chain. This would be a variant of Plan S.
I think this is plausibly as good or better than Plan A, not sure. One issue with it is that a covert project with, say, 100k GPUs doesn’t really confer much geostrategic advantage in Plan A, but in Plan S, it might. Imagine: It’s 2040. The economy has recovered from the compute supply chain being dismantled; people have learned to live without computers. But negotiations for how to restart AI progress safely and transparently and in a power-distributed way are dragging on and on and it seems like it’s basically never going to reach agreement. Meanwhile, the covert project has managed to make an OOM or two of algorithmic progress since 2030, and is just a few years away from fully automating AI R&D, at which point they’ll probably have ASI within a few years of that...
Idk. We have a model of AI progress + model of black sites that tries to model situations like this. I don’t think it’s obvious either way how it would go.
My other objection is that I feel a lot of despair when thinking in near-mode about the Plan A proposal of slowing down algorithmic progress.
----
India’s leading company publishes a paper about a training dataset they used to make the user-experience for their AI smoother. The Indian regulator apparently green-lighted it, but they are known for their lenient approach. Some academics notice that the Indian model is now not just smoother to use, but it subtly feels like it’s smarter than other models, even though it doesn’t directly show on the standard benchmarks. The American AISI starts getting emails from academics, explaining that they feel like the Indian model got smarter, and that they think the paper the Indians published was too high-level to fully reconstruct what they did.
Meanwhile the American AISI is also getting emails from academics complaining about how the European AI is too biased against minorities, and how the Brazilian AI has too high persuasive capabilities, and how the compute cluster in the ocean is harming the fish. They are also tasked with defending the US companies from the totally unfair complaints coming from China and India that they are not transparent enough about their research.
The American AISI only has 200 people at this point, because they started out small, and you can’t easily grow an organization more than 2x per year. Also, 50 out of the 200 people joined because they care about the ocean cluster harming the fish. (It is true though that probably it helps a bunch that they have good AI assistants. Maybe the crux is that I don’t believe it that much that the AI assistants will solve the dysfunctionality.)
The American AISI starts a bilateral conversation with the Indian regulators, but by the time they get to anywhere, it looks like the Europeans and two American companies have already probably copied something like what the Indians did. The world clearly didn’t end, and the AIs are just a bit smoother to use now. So no one escalates to a big diplomatic fight, and they quietly accept that the Indians didn’t publish enough details this time. Everyone keeps scaling up the technique with more compute and more data.
A month later, there is a new report which indicates that some interpretability techniques show that some scheming propensities are maybe going up, maybe linked to the new technique. After another month (do you know how slow it is to do anything in a government agency?) the American AISI calls up the President, and tells him to put pressure on American companies and other world leaders to roll back the new technique. Unfortunately, the President doesn’t have much time, because the Australian PM is pressuring him to make sure no one’s AI goes along with human rights violations in an ongoing war in the Middle East. Plus, he needs to speak at an environmentalist conference about the fish. (Also, he is still the President, running the non-AI aspects of the country.)
Now the President needs to call up the other world leaders and make it clear that he is willing to go to war if needed, because this interp scheming score really shouldn’t go up (unlike last time when a bias score was going up, but after a lot of yelling, people decided it’s not worth fighting over). So the President and other world leaders (reminder: Xi is 73, Modi is 75, Trump is 80, Biden left office at 82) get together and discuss the interp results and come to an agreement.
----
I think probably the sheer amount of bureaucracy still slows down algorithmic progress in the companies, and probably the government agencies pushing for safer directions and trying to ban more unsafe directions will have a non-zero correlation with actual safety. But it’s going to be a mess, and I don’t think you can slow down progress more than 2x this way, and the companies and countries that are more willing to skirt the rules will come out ahead more.
In contrast, GPUs, fabs and EUV machines are physical objects. I have much more trust in the government to competently control them and slow progress that way than through governing algorithmic progress.
I think I’m just not as afraid of covert projects. In the world that has enough political will for Plan A or compute supply dismantling, I think it’s likely that neither the US nor China even wants to attempt a covert project, because they are both genuinely afraid of AI. And I don’t see who else could attempt a covert project with 100k hidden GPUs.
And once the compute chain dismantling happens, it becomes very obvious that the leading powers are extremely serious about not wanting AI. I think it’s over 70% likely that espionage will uncover the existence of a covert project, and then they will very likely get bombed in the world where everyone was so serious that they already blew up their own EUV machines. I expect that the US and China will be aware of this and not attempt a covert project.
(It’s also my understanding that GPUs probably mostly break after about 5 years of use. That will really hurt the covert project if they can’t source new GPUs, no? I don’t remember this being addressed in the supplement, but maybe I missed something.)
Also, even if a covert project manages to build something smarter than anyone else has, that’s not necessarily catastrophic. If they manage to build such a strong superintelligence that it can build invincible nanobot-armies in a basement without any previous industrial build-out, that’s bad. But if they would just want to do the usual strategy of building robots to build better robots to build more compute and drone armies, that will very likely be visible an be noticed, in a world that otherwise doesn’t have advanced compute an robotics, and will lead to bombing.
My understanding is that in this way, Plan A is more vulnerable to covert projects than the compute dismantling plan: in Plan A, if someone secretly builds a smarter AI than anyone else has, they can use it to devise a plan to bring a few percent of the huge compute clusters outside the easily bombable places, and then hook up the secretly built smart AI to the compute and the already existing huge robot economy and drone army. The path to victory seems much harder in the world without a lot of compute and robots.
The game theory doesn’t work out this way. In two player models, the US and China can always each get a geopolitical power advantage from developing an AI more powerful than socially optimal, and so have an incentive to defect.
If they believe AI is very dangerous, they will just set the treaty capability cap lower, and still defect on it.
I think countries sometimes have honor and follow their commitments even if it’s not locally game theoretically optimal.
Sure, but it’s worth noting that’s a different reason. IMO fear of AI sets the perceived socially optimal outcome, and honor, collusion, transparency, etc determine whether a treaty holds.
.
Your proposal would be net positive but the question is whether it’s better than Plan A. Even if takeoff is very slow post-deal if you didn’t advance ai capabilities much during the deal or use them heavily to do useful work then you get much less benefit from the deal than in Plan A.
I think it’s pretty likely that if you start a very serious pause, with compute chain dismantling, then the race doesn’t restart for a very long time. And in the meantime, the world might just become a better place by default. I find it relatively plausible that in the next hundred years, the world would make as much progress towards democracy, stability and peace as Europe made since 1926. If US and China and the other great powers of 2126 will be on as friendly terms as France and Germany are now, then I will feel much better about slowly restarting AI development. If you think that the deal is very likely to break early, or if you don’t believe that the default secular trends in the world point in a good direction, then admittedly this less promising.
See https://ai-2040.com/supplements/deal-decline for my probabilistic estimates here.
I think it should be if people were well-calibrated? (Which is more likely in worlds where people are taking things seriously enough for these proposals.)
Delaying ~infinity growth by 100 years seems more than twice as bad as delaying ~infinity growth for 10 years. With exponential discounting, the only way I can currently see for this to be false (though there’s probably other ways) is if you think that most of the value of “~infinity growth now” would already have been lost by moving to “~infinity growth in 10 years”. But that doesn’t seem consistent with how low real interest rates are. (They tend to value stuff in 10 years at more than half as much as stuff now.)
And plan A looks better than “singularity in 10 years” by any very time-sensitive financial measure, since you get lots of growth in the intermediate 10 years too.
Yes, that’s fair. But given that Plan A starts with a temporary full shutdown, and it will be generally hard to know what pace of progress will be allowed later, I think markets will probably initially think that there is substantial chance that the temporary pause won’t really be lifted or progress will be reversed in other ways. (E.g. I don’t expect markets to be enthusiastic about the proposal to put all data centers in Mongolia, under the threat of Chinese bombing.) So I think the initial stock market crash at the announcement of Plan A will still be huge—I would now guess between 30% and 50% as big as in the case of full dismantling of the compute supply chain.
I think one of the main political barriers to starting either the compute dismantling or Plan A is the unpopular initial crash and the corresponding industry outcry, and that will comparably be huge in both cases.
Sure. Though conversely, there should also be a big probability that the pause will be immimently lifted and progress will keep going just as before. Whereas there’s much less hope of that if there’s an announcement to destroy the compute supply chain.
I suppose that if the initial announcement comes with a 30% probability that all compute will be destroyed, then the crash should probably be at least 30% as large as a definite announcement that the compute will be destroyed? Though 30% on that given the initial pause seems maybe 2x too high to me or something.