This is a good analysis of the geopolitical situation but some claims seem very overconfident. For example:
The only way to avoid these pressures is if we could build common knowledge, at any given time, that no one is trying to develop ASI.
It is very possible one lab/country gains a decisive advantage before any others. The approach to ASI Is likely to be chaotic and fraught with disagreement. It very well may not be obvious what is happening to other powers until a decisive advantage is gained. If the lab/country with a decisive advantage succeeds at technical alignment you may end up in a world which bypasses many of these concerns.
a foreign government will take over your project, or at least steal all the progress you’ve made so far and then pour their resources into going faster than you.
It took the Soviets years of espionage to steal atomic secrets. If a lab is approaching ASI, one can expect that the pre-ASI AI will be heavily woven into their security architecture for securing model weights which may effectively prevent nation states from stealing them.
Overall your post was quite high quality and explores a neglected area of AI discourse (And I agree we should pause AI!) but I also don’t think things are quite as hopeless as outlined.
If AI development is more insight-driven than compute-driven, then there is more room for sudden progress that gains a decisive advantage over other labs and govs before getting noticed (other entities suspecting the lab getting close to ASI with non-neglible confidence) and reacting. This allows the lab to control the singleton instead of the mainstream labs and govs, and in this situation, the lab might escape from race dynamics.
However, this scenario results in a random lab controlling a singleton. While it’s not as hopeless as a singleton built by a racing entity, it doesn’t look really hopeful either.
If we’re talking about the differential effect of a given lab joining the race, then they could have a positive effect, if we know they have good intentions to benefit humanity. However, it’s still difficult to ensure the good intentions are still there when they actually get to ASI.
I think there is an interesting middle ground here too. You have the dichotomy Insight versus compute driven.
I would generally agree with your assessment.
Within the compute driven paradigm I’d propose a sub delineation. Intelligence able to generate novel insights versus intelligence able to succeed only in highly specified domains.
It is possible that LLM intelligence only scales in verifiable domains, but not in “insight” driven domains where you end up with a an AI that can solve math problems, identify vulnerabilities, and code in a vastly superhuman way, but is notably hobbled in other domains.
If we’re talking about the differential effect of a given lab joining the race, then they could have a positive effect, if we know they have good intentions to benefit humanity.
FWIW, I think that this has mostly the effect of just adding fuel to the fire, because the government takes over the project regardless of the intentions of the company.
For this to be different, the “insight” would have to accelerate progress to ASI so much that the company can build ASI in a very short time while staying under the radar, including having very few employees and using little compute.
If you buy my arguments in the section “Why Technical AI safety agendas do not address this problem”, the appearance of such an insight would actually be extremely bad: AI safety is always more bottlenecked on humans compared to capabilities, and this company has very few humans!
Inventing God while no one’s watching? Are labs likely to keep a decisive advantage sufficiently obscure long enough to take over the world? We saw with Mythos that Anthropic announced it very loudly, in close consultation with the US government. We have an executive order that “voluntarily” requires labs to let the US government to review models before release even at this level of capability.
But even if one entity gets the decisive advantage, we’re still just back in the “nightmare singleton” scenario.
Separately, “succeeds at technical alignment” here has to mean that technical alignment is so easy, you can do it under competitive pressures. Do even the “alignment is easy” folk believe that? Or do you mean that the singleton takes over first, then solves alignment? It doesn’t seem like you take over the world and still remain in control with a misaligned ASI...
Would pre-ASI AI be enough to stop every point of espionage, or at least enough of them? Anthropic is still at least a year from SL-4 - not even 5 - and who knows how the other labs are doing. Can it stop espionage that isn’t purely cyber? Do we think other countries don’t have spies in at least one of the major labs? Are AI secrets as well protected as nuclear ones were?
Ultimately, even granting that it could be physically possible that a) you could get a surprise decisive advantage, b) solve technical alignment before then, c) ward off all espionage attempts before then, and d) the resulting singleton acts to humanity’s best interests—is the conjunction of all those a high enough probability to be termed “not quite as hopeless”?
Aside from whether the claims you are citing are true or well-calibrated, I want to point something out.
I actually think that your example scenarios here illustrate exactly the type of scenario that I wanted to disarm with this post. Or if not disarm them, at least give people the tools they need to disarm them.
It is very possible one lab/country gains a decisive advantage before any others. The approach to ASI Is likely to be chaotic and fraught with disagreement. It very well may not be obvious what is happening to other powers until a decisive advantage is gained. If the lab/country with a decisive advantage succeeds at technical alignment you may end up in a world which bypasses many of these concerns.
This very specific scenario, that I think could technically happen but is extremely unlikely, falls to the third filter, “nightmare singleton”: the winning AI company having to create a singleton in order to actually bypass the first and second filters.
You may believe this can be a good outcome, but you are still fundamentally trusting that the singleton established by a private company will be good for you / most people, which I don’t qualify as a satisfying solution to the third filter.
(The reason I think this scenario is unlikely is that for this to work the AI company would’ve had to get a really, really big advantage over anyone else: how did they do this without cutting too many corners on safety and failing at the second filter? And of course, the thing about the government taking over the project.)
It took the Soviets years of espionage to steal atomic secrets. If a lab is approaching ASI, one can expect that the pre-ASI AI will be heavily woven into their security architecture for securing model weights which may effectively prevent nation states from stealing them.
I could spend time arguing why I think this scenario is unlikely, but I think this would miss the main point of the post: it doesn’t address the fact that the winning AI company still needs to make a singleton, and we’re failing at the third filter.
This is a good analysis of the geopolitical situation but some claims seem very overconfident. For example:
The only way to avoid these pressures is if we could build common knowledge, at any given time, that no one is trying to develop ASI.
It is very possible one lab/country gains a decisive advantage before any others. The approach to ASI Is likely to be chaotic and fraught with disagreement. It very well may not be obvious what is happening to other powers until a decisive advantage is gained. If the lab/country with a decisive advantage succeeds at technical alignment you may end up in a world which bypasses many of these concerns.
a foreign government will take over your project, or at least steal all the progress you’ve made so far and then pour their resources into going faster than you.
It took the Soviets years of espionage to steal atomic secrets. If a lab is approaching ASI, one can expect that the pre-ASI AI will be heavily woven into their security architecture for securing model weights which may effectively prevent nation states from stealing them.
Overall your post was quite high quality and explores a neglected area of AI discourse (And I agree we should pause AI!) but I also don’t think things are quite as hopeless as outlined.
Partially agree.
If AI development is more insight-driven than compute-driven, then there is more room for sudden progress that gains a decisive advantage over other labs and govs before getting noticed (other entities suspecting the lab getting close to ASI with non-neglible confidence) and reacting. This allows the lab to control the singleton instead of the mainstream labs and govs, and in this situation, the lab might escape from race dynamics.
However, this scenario results in a random lab controlling a singleton. While it’s not as hopeless as a singleton built by a racing entity, it doesn’t look really hopeful either.
If we’re talking about the differential effect of a given lab joining the race, then they could have a positive effect, if we know they have good intentions to benefit humanity. However, it’s still difficult to ensure the good intentions are still there when they actually get to ASI.
I think there is an interesting middle ground here too. You have the dichotomy Insight versus compute driven.
I would generally agree with your assessment.
Within the compute driven paradigm I’d propose a sub delineation. Intelligence able to generate novel insights versus intelligence able to succeed only in highly specified domains.
It is possible that LLM intelligence only scales in verifiable domains, but not in “insight” driven domains where you end up with a an AI that can solve math problems, identify vulnerabilities, and code in a vastly superhuman way, but is notably hobbled in other domains.
FWIW, I think that this has mostly the effect of just adding fuel to the fire, because the government takes over the project regardless of the intentions of the company.
For this to be different, the “insight” would have to accelerate progress to ASI so much that the company can build ASI in a very short time while staying under the radar, including having very few employees and using little compute.
If you buy my arguments in the section “Why Technical AI safety agendas do not address this problem”, the appearance of such an insight would actually be extremely bad: AI safety is always more bottlenecked on humans compared to capabilities, and this company has very few humans!
Inventing God while no one’s watching? Are labs likely to keep a decisive advantage sufficiently obscure long enough to take over the world? We saw with Mythos that Anthropic announced it very loudly, in close consultation with the US government. We have an executive order that “voluntarily” requires labs to let the US government to review models before release even at this level of capability.
But even if one entity gets the decisive advantage, we’re still just back in the “nightmare singleton” scenario.
Separately, “succeeds at technical alignment” here has to mean that technical alignment is so easy, you can do it under competitive pressures. Do even the “alignment is easy” folk believe that? Or do you mean that the singleton takes over first, then solves alignment? It doesn’t seem like you take over the world and still remain in control with a misaligned ASI...
Would pre-ASI AI be enough to stop every point of espionage, or at least enough of them? Anthropic is still at least a year from SL-4 - not even 5 - and who knows how the other labs are doing. Can it stop espionage that isn’t purely cyber? Do we think other countries don’t have spies in at least one of the major labs? Are AI secrets as well protected as nuclear ones were?
Ultimately, even granting that it could be physically possible that a) you could get a surprise decisive advantage, b) solve technical alignment before then, c) ward off all espionage attempts before then, and d) the resulting singleton acts to humanity’s best interests—is the conjunction of all those a high enough probability to be termed “not quite as hopeless”?
Aside from whether the claims you are citing are true or well-calibrated, I want to point something out.
I actually think that your example scenarios here illustrate exactly the type of scenario that I wanted to disarm with this post. Or if not disarm them, at least give people the tools they need to disarm them.
This very specific scenario, that I think could technically happen but is extremely unlikely, falls to the third filter, “nightmare singleton”: the winning AI company having to create a singleton in order to actually bypass the first and second filters.
You may believe this can be a good outcome, but you are still fundamentally trusting that the singleton established by a private company will be good for you / most people, which I don’t qualify as a satisfying solution to the third filter.
(The reason I think this scenario is unlikely is that for this to work the AI company would’ve had to get a really, really big advantage over anyone else: how did they do this without cutting too many corners on safety and failing at the second filter? And of course, the thing about the government taking over the project.)
I could spend time arguing why I think this scenario is unlikely, but I think this would miss the main point of the post: it doesn’t address the fact that the winning AI company still needs to make a singleton, and we’re failing at the third filter.