If they have the political will to do Plan A, they very well might also have the political will to dismantle the compute supply chain. This would be a variant of Plan S.
I think this is plausibly as good or better than Plan A, not sure. One issue with it is that a covert project with, say, 100k GPUs doesn’t really confer much geostrategic advantage in Plan A, but in Plan S, it might. Imagine: It’s 2040. The economy has recovered from the compute supply chain being dismantled; people have learned to live without computers. But negotiations for how to restart AI progress safely and transparently and in a power-distributed way are dragging on and on and it seems like it’s basically never going to reach agreement. Meanwhile, the covert project has managed to make an OOM or two of algorithmic progress since 2030, and is just a few years away from fully automating AI R&D, at which point they’ll probably have ASI within a few years of that...
Idk. We have a model of AI progress + model of black sites that tries to model situations like this. I don’t think it’s obvious either way how it would go.
My other objection is that I feel a lot of despair when thinking in near-mode about the Plan A proposal of slowing down algorithmic progress.
----
India’s leading company publishes a paper about a training dataset they used to make the user-experience for their AI smoother. The Indian regulator apparently green-lighted it, but they are known for their lenient approach. Some academics notice that the Indian model is now not just smoother to use, but it subtly feels like it’s smarter than other models, even though it doesn’t directly show on the standard benchmarks. The American AISI starts getting emails from academics, explaining that they feel like the Indian model got smarter, and that they think the paper the Indians published was too high-level to fully reconstruct what they did.
Meanwhile the American AISI is also getting emails from academics complaining about how the European AI is too biased against minorities, and how the Brazilian AI has too high persuasive capabilities, and how the compute cluster in the ocean is harming the fish. They are also tasked with defending the US companies from the totally unfair complaints coming from China and India that they are not transparent enough about their research.
The American AISI only has 200 people at this point, because they started out small, and you can’t easily grow an organization more than 2x per year. Also, 50 out of the 200 people joined because they care about the ocean cluster harming the fish. (It is true though that probably it helps a bunch that they have good AI assistants. Maybe the crux is that I don’t believe it that much that the AI assistants will solve the dysfunctionality.)
The American AISI starts a bilateral conversation with the Indian regulators, but by the time they get to anywhere, it looks like the Europeans and two American companies have already probably copied something like what the Indians did. The world clearly didn’t end, and the AIs are just a bit smoother to use now. So no one escalates to a big diplomatic fight, and they quietly accept that the Indians didn’t publish enough details this time. Everyone keeps scaling up the technique with more compute and more data.
A month later, there is a new report which indicates that some interpretability techniques show that some scheming propensities are maybe going up, maybe linked to the new technique. After another month (do you know how slow it is to do anything in a government agency?) the American AISI calls up the President, and tells him to put pressure on American companies and other world leaders to roll back the new technique. Unfortunately, the President doesn’t have much time, because the Australian PM is pressuring him to make sure no one’s AI goes along with human rights violations in an ongoing war in the Middle East. Plus, he needs to speak at an environmentalist conference about the fish. (Also, he is still the President, running the non-AI aspects of the country.)
Now the President needs to call up the other world leaders and make it clear that he is willing to go to war if needed, because this interp scheming score really shouldn’t go up (unlike last time when a bias score was going up, but after a lot of yelling, people decided it’s not worth fighting over). So the President and other world leaders (reminder: Xi is 73, Modi is 75, Trump is 80, Biden left office at 82) get together and discuss the interp results and come to an agreement.
----
I think probably the sheer amount of bureaucracy still slows down algorithmic progress in the companies, and probably the government agencies pushing for safer directions and trying to ban more unsafe directions will have a non-zero correlation with actual safety. But it’s going to be a mess, and I don’t think you can slow down progress more than 2x this way, and the companies and countries that are more willing to skirt the rules will come out ahead more.
In contrast, GPUs, fabs and EUV machines are physical objects. I have much more trust in the government to competently control them and slow progress that way than through governing algorithmic progress.
I think I’m just not as afraid of covert projects. In the world that has enough political will for Plan A or compute supply dismantling, I think it’s likely that neither the US nor China even wants to attempt a covert project, because they are both genuinely afraid of AI. And I don’t see who else could attempt a covert project with 100k hidden GPUs.
And once the compute chain dismantling happens, it becomes very obvious that the leading powers are extremely serious about not wanting AI. I think it’s over 70% likely that espionage will uncover the existence of a covert project, and then they will very likely get bombed in the world where everyone was so serious that they already blew up their own EUV machines. I expect that the US and China will be aware of this and not attempt a covert project.
(It’s also my understanding that GPUs probably mostly break after about 5 years of use. That will really hurt the covert project if they can’t source new GPUs, no? I don’t remember this being addressed in the supplement, but maybe I missed something.)
Also, even if a covert project manages to build something smarter than anyone else has, that’s not necessarily catastrophic. If they manage to build such a strong superintelligence that it can build invincible nanobot-armies in a basement without any previous industrial build-out, that’s bad. But if they would just want to do the usual strategy of building robots to build better robots to build more compute and drone armies, that will very likely be visible an be noticed, in a world that otherwise doesn’t have advanced compute an robotics, and will lead to bombing.
My understanding is that in this way, Plan A is more vulnerable to covert projects than the compute dismantling plan: in Plan A, if someone secretly builds a smarter AI than anyone else has, they can use it to devise a plan to bring a few percent of the huge compute clusters outside the easily bombable places, and then hook up the secretly built smart AI to the compute and the already existing huge robot economy and drone army. The path to victory seems much harder in the world without a lot of compute and robots.
In the world that has enough political will for Plan A or compute supply dismantling, I think it’s likely that neither the US nor China even wants to attempt a covert project, because they are both genuinely afraid of AI.
The game theory doesn’t work out this way. In two player models, the US and China can always each get a geopolitical power advantage from developing an AI more powerful than socially optimal, and so have an incentive to defect.
If they believe AI is very dangerous, they will just set the treaty capability cap lower, and still defect on it.
Sure, but it’s worth noting that’s a different reason. IMO fear of AI sets the perceived socially optimal outcome, and honor, collusion, transparency, etc determine whether a treaty holds.
If they have the political will to do Plan A, they very well might also have the political will to dismantle the compute supply chain. This would be a variant of Plan S.
I think this is plausibly as good or better than Plan A, not sure. One issue with it is that a covert project with, say, 100k GPUs doesn’t really confer much geostrategic advantage in Plan A, but in Plan S, it might. Imagine: It’s 2040. The economy has recovered from the compute supply chain being dismantled; people have learned to live without computers. But negotiations for how to restart AI progress safely and transparently and in a power-distributed way are dragging on and on and it seems like it’s basically never going to reach agreement. Meanwhile, the covert project has managed to make an OOM or two of algorithmic progress since 2030, and is just a few years away from fully automating AI R&D, at which point they’ll probably have ASI within a few years of that...
Idk. We have a model of AI progress + model of black sites that tries to model situations like this. I don’t think it’s obvious either way how it would go.
My other objection is that I feel a lot of despair when thinking in near-mode about the Plan A proposal of slowing down algorithmic progress.
----
India’s leading company publishes a paper about a training dataset they used to make the user-experience for their AI smoother. The Indian regulator apparently green-lighted it, but they are known for their lenient approach. Some academics notice that the Indian model is now not just smoother to use, but it subtly feels like it’s smarter than other models, even though it doesn’t directly show on the standard benchmarks. The American AISI starts getting emails from academics, explaining that they feel like the Indian model got smarter, and that they think the paper the Indians published was too high-level to fully reconstruct what they did.
Meanwhile the American AISI is also getting emails from academics complaining about how the European AI is too biased against minorities, and how the Brazilian AI has too high persuasive capabilities, and how the compute cluster in the ocean is harming the fish. They are also tasked with defending the US companies from the totally unfair complaints coming from China and India that they are not transparent enough about their research.
The American AISI only has 200 people at this point, because they started out small, and you can’t easily grow an organization more than 2x per year. Also, 50 out of the 200 people joined because they care about the ocean cluster harming the fish. (It is true though that probably it helps a bunch that they have good AI assistants. Maybe the crux is that I don’t believe it that much that the AI assistants will solve the dysfunctionality.)
The American AISI starts a bilateral conversation with the Indian regulators, but by the time they get to anywhere, it looks like the Europeans and two American companies have already probably copied something like what the Indians did. The world clearly didn’t end, and the AIs are just a bit smoother to use now. So no one escalates to a big diplomatic fight, and they quietly accept that the Indians didn’t publish enough details this time. Everyone keeps scaling up the technique with more compute and more data.
A month later, there is a new report which indicates that some interpretability techniques show that some scheming propensities are maybe going up, maybe linked to the new technique. After another month (do you know how slow it is to do anything in a government agency?) the American AISI calls up the President, and tells him to put pressure on American companies and other world leaders to roll back the new technique. Unfortunately, the President doesn’t have much time, because the Australian PM is pressuring him to make sure no one’s AI goes along with human rights violations in an ongoing war in the Middle East. Plus, he needs to speak at an environmentalist conference about the fish. (Also, he is still the President, running the non-AI aspects of the country.)
Now the President needs to call up the other world leaders and make it clear that he is willing to go to war if needed, because this interp scheming score really shouldn’t go up (unlike last time when a bias score was going up, but after a lot of yelling, people decided it’s not worth fighting over). So the President and other world leaders (reminder: Xi is 73, Modi is 75, Trump is 80, Biden left office at 82) get together and discuss the interp results and come to an agreement.
----
I think probably the sheer amount of bureaucracy still slows down algorithmic progress in the companies, and probably the government agencies pushing for safer directions and trying to ban more unsafe directions will have a non-zero correlation with actual safety. But it’s going to be a mess, and I don’t think you can slow down progress more than 2x this way, and the companies and countries that are more willing to skirt the rules will come out ahead more.
In contrast, GPUs, fabs and EUV machines are physical objects. I have much more trust in the government to competently control them and slow progress that way than through governing algorithmic progress.
I think I’m just not as afraid of covert projects. In the world that has enough political will for Plan A or compute supply dismantling, I think it’s likely that neither the US nor China even wants to attempt a covert project, because they are both genuinely afraid of AI. And I don’t see who else could attempt a covert project with 100k hidden GPUs.
And once the compute chain dismantling happens, it becomes very obvious that the leading powers are extremely serious about not wanting AI. I think it’s over 70% likely that espionage will uncover the existence of a covert project, and then they will very likely get bombed in the world where everyone was so serious that they already blew up their own EUV machines. I expect that the US and China will be aware of this and not attempt a covert project.
(It’s also my understanding that GPUs probably mostly break after about 5 years of use. That will really hurt the covert project if they can’t source new GPUs, no? I don’t remember this being addressed in the supplement, but maybe I missed something.)
Also, even if a covert project manages to build something smarter than anyone else has, that’s not necessarily catastrophic. If they manage to build such a strong superintelligence that it can build invincible nanobot-armies in a basement without any previous industrial build-out, that’s bad. But if they would just want to do the usual strategy of building robots to build better robots to build more compute and drone armies, that will very likely be visible an be noticed, in a world that otherwise doesn’t have advanced compute an robotics, and will lead to bombing.
My understanding is that in this way, Plan A is more vulnerable to covert projects than the compute dismantling plan: in Plan A, if someone secretly builds a smarter AI than anyone else has, they can use it to devise a plan to bring a few percent of the huge compute clusters outside the easily bombable places, and then hook up the secretly built smart AI to the compute and the already existing huge robot economy and drone army. The path to victory seems much harder in the world without a lot of compute and robots.
The game theory doesn’t work out this way. In two player models, the US and China can always each get a geopolitical power advantage from developing an AI more powerful than socially optimal, and so have an incentive to defect.
If they believe AI is very dangerous, they will just set the treaty capability cap lower, and still defect on it.
I think countries sometimes have honor and follow their commitments even if it’s not locally game theoretically optimal.
Sure, but it’s worth noting that’s a different reason. IMO fear of AI sets the perceived socially optimal outcome, and honor, collusion, transparency, etc determine whether a treaty holds.
.