Why is it worse for x risk for China to win the AI race?
My understanding of the standard threat model is that, at some point, governments will need to step in and shut down or take control over profitable and popular projects for the good of all society. I look at China, and I look at the US, and I can’t say “the US is the country I would bet on to hit the big red button here”.
In my experience, when people say “it’s worse for China to win the AI race than America”, their main concern is that Chinese control of the far future would lead to a much less valuable future than American control would, not that American control reduces P(AI takeover). E.g. see this comment.
FWIW, I have seen a decent amount of flip-flopping on this question. My current guess is that most of the time when people say this, they don’t mean either of those things but have some other reason for the belief, and choose the justification that they think will be most likely compelling to their interlocutor (like, I’ve had a bunch of instances of the same person telling me at different times that they were centrally concerned about China because it increased P(AI takeover) and then at a different point in time in a different social context that they were centrally concerned about Chinese values being less good by their lights if optimized).
FWIW, my enthusiasm for “make America more good at AI than China” type policies comes somewhat more from considerations like “a larger US advantage lets the US spend more of a lead on safety without needing international cooperation” than considerations like “a CCP-led corrigible ASI would lead to much worse outcomes than a USG-led corrigible ASI”. Though both are substantial factors for me and I’m fairly uncertain; I would not be surprised if my ordering here switched in 6 months.
FWIW, my view is that the badness is somewhat evenly split between increases to takeover risk and the far future being worse conditional on no misaligned AI takeover. (Maybe 2⁄5 increases to misaligned AI takeover risk and 3⁄5 far future being worse? It depend on what you mean though because China winning is also correlated with US/China being close which is also probably correlated with more racing and thus probably more misaligned AI takeover risk?)
the far future being worse conditional on no takeover
To clarify, by “takeover” here do you mean “misaligned AI takeover”? I.e. does your “no takeover” conditional include worlds where e.g. the CCP uses AI to takeover?
I can think of plenty of reasons, of varying levels of sensibility.
Arguments
Some people believe that (a) controlled on-paradigm ASI is possible, but that (b) it would require spending some nontrivial amount of resources/time on alignment/control research[1], and that (c) the US AGI labs are much more likely to do it than the Chinese ones. Therefore, the US winning is less likely to lead to omnicide.
I think it’s not unreasonable to believe (c), so if you believe (a) and (b), as many people do, the conclusion checks out. I assign low (but nonzero) probability to (a), though.
Even if the Chinese labs can keep ASI aligned/under control, some people are scared of being enslaved by the CCP, and think that the USG becoming god is going to be better for them.[2] This probably includes people who profess to only care about the nobody-should-build-it thing: they un/semi-consciously track the S-risk possibility, and it’s awful-feeling enough to affect their thinking even if they assign it low probability.
I think that’s a legitimate worry; S-risks are massively worse than X-risks. But I don’t expect the USG’s apotheosis to look pretty either, especially not under the current administration, and same for the apotheosis of most AGI labs, so the point is mostly moot.
I guess Anthropic or maybe DeepMind could choose non-awful results? So sure, if the current paradigm can lead to controlled ASI, and the USG stays asleep, and Anthropic/DM are the favorites to win, “make China lose” has some sense.
Variant on the above scenarios, but which does involve an international pause, with some coordination to only develop ASI once it can be kept under control. This doesn’t necessarily guarantee that the ASI, once developed, will be eudaimonic, so “who gets to ASI first/has more say on ASI” may matter; GOTO (2).
The AI-Risk advocates may feel that they have more influence on the leadership of the US labs. For US-based advocates, this is almost certainly correct. If that leadership can be convinced to pause, this buys us as much time as it’d take for the runners-up to catch up. Thus, the further behind China is, the more time we can buy in this hypothetical.
In addition, if China is way behind, it’s more likely that the US AGI labs would agree to stop, since more time to work would increase the chances of success of [whatever we want the pause for, e. g. doing alignment research or trying to cause an international ban].
Same as (4), but for governments. Perhaps the USG is easier to influence into arguing for an international pause. If so, both (a) the USG is more likely to do this if it feels that it’s comfortably ahead of China rather than nose-to-nose, (b) China is more likely to agree to an international ban if the USG is speaking from a position of power and is ahead on AI than if it’s behind/nose-to-nose. (Both because the ban would be favorable to China geopolitically, and because the X-risk arguments would sound more convincing if they don’t look like motivated reasoning/bullshit you’re inventing to convince China to abandon a technology that gives it geopolitical lead on the US.)
Some less sensible/well-thought-out variants of the above, e. g.:
Having the illusion of having more control over the US labs/government.
Semi/un-consciously feeling that it’d be better if your nation ends the world than if the Chinese do it.
Semi/un-consciously feeling that it’d be better if your nation is more powerful/ahead of a foreign one, independent of any X-risk considerations.
Suppose you think the current paradigm doesn’t scale to ASI, or that we’ll succeed in internationally banning ASI research. The amount of compute at a nation’s disposal is likely to still be increasingly important in the coming future (just because it’d allow to better harness the existing AI technology, for military and economic ends). Thus, constricting China’s access is likely to be better for the US as well.
This has nothing to do with X-risks though, it’s prosaic natsec stuff.
tl;dr:
If we get alignment by default, some US-based actors winning may be more likely to lead to a good future than the Chinese actors winning.
If on-paradigm ASI alignment is possible given some low-but-nontrivial resource expenditure, the US labs may be more likely to spend the resources on it than the Chinese ones.
US AI Safety advocates may have more control over the US AGI labs and/or the USG. The more powerful those are relative to the foreign AGI researchers, the more leverage that influence provides, including for slowing down/banning AGI research.
US AI Safety advocates may be at least partly motivated by dumb instincts for “my nation good, their nation bad”, and therefore want the US to win even if it’s winning a race-to-suicide.
Keeping a compute lead may be geopolitically important even in non-ASI worlds.
E. g., Ryan Greenblatt thinks that spending just 5% more resources than is myopically commercially expedient would drive the risk down to 50%. AI 2027 also assumes something like this.
E. g., Ryan Greenblatt thinks that spending 5% more resources than is myopically commercially expedient would be enough. AI 2027 also assumes something like this.
TBC, my view isn’t that this is sufficient for avoiding takeover risk, it is that this suffices for “you [to] have a reasonable chance of avoiding AI takeover (maybe 50% chance of misaligned AI takeover?)”.
(You seem to understand that this is my perspective and I think this is also mostly clear from the context in the box, but I wanted to clarify this given the footnote might be read in isolation or misinterpreted.)
There are some additional reasons, beyond the question of which values would be embedded in the AGI systems, to not prefer AGI development in China, that I haven’t seen mentioned here:
Systemic opacity, state-driven censorship, and state control of the media means AGI development under direct or indirect CCP control would probably be less transparent than in the US, and the world may be less likely to learn about warning shots, wrongheaded decisions, reckless behaviour, etc. True, there was the Manhattan Project, but that was quite long ago; recent examples like the CCP’s suppression of information related to the origins of COVID feel more salient and relevant.
There are more checks and balances in the US than in China, which you may think could e.g., positively influence regulation; or if there’s a government project, help incentivise responsible decisions there; or if someone attempts to concentrate power using some early AGI, stop that from happening. E.g., in the West voters have some degree of influence over the government, there’s the free press, the judiciary, an ecosystem of nonprofits, and so on. In China, the CCP doesn’t have total control, but much more so than Western governments do.
I think it’s also very rare that people are actually faced with a choice between “AGI in the US” versus “AGI in China”. A more accurate but still flawed model of the choice people are sometimes faced with is “AGI in the US” versus “AGI in the US and in China”, or even “AGI in the US, and in China 6-12 months later” versus “AGI in the US, and in China 3-6 months later”.
I don’t necessarily disagree, but one big thing is freedom of speech. If the party line is to go big on AI, which is likely, given current investments, I’m not betting on Chinese Yudkowsky. Same with frontier lab whistleblowers, etc
Why is it worse for x risk for China to win the AI race?
My understanding of the standard threat model is that, at some point, governments will need to step in and shut down or take control over profitable and popular projects for the good of all society. I look at China, and I look at the US, and I can’t say “the US is the country I would bet on to hit the big red button here”.
There’s got to be something I’m missing here.
In my experience, when people say “it’s worse for China to win the AI race than America”, their main concern is that Chinese control of the far future would lead to a much less valuable future than American control would, not that American control reduces P(AI takeover). E.g. see this comment.
FWIW, I have seen a decent amount of flip-flopping on this question. My current guess is that most of the time when people say this, they don’t mean either of those things but have some other reason for the belief, and choose the justification that they think will be most likely compelling to their interlocutor (like, I’ve had a bunch of instances of the same person telling me at different times that they were centrally concerned about China because it increased P(AI takeover) and then at a different point in time in a different social context that they were centrally concerned about Chinese values being less good by their lights if optimized).
It really depends on what you mean by “most of the time when people say this”. I don’t think my experience matches yours.
FWIW, my enthusiasm for “make America more good at AI than China” type policies comes somewhat more from considerations like “a larger US advantage lets the US spend more of a lead on safety without needing international cooperation” than considerations like “a CCP-led corrigible ASI would lead to much worse outcomes than a USG-led corrigible ASI”. Though both are substantial factors for me and I’m fairly uncertain; I would not be surprised if my ordering here switched in 6 months.
FWIW, my view is that the badness is somewhat evenly split between increases to takeover risk and the far future being worse conditional on no misaligned AI takeover. (Maybe 2⁄5 increases to misaligned AI takeover risk and 3⁄5 far future being worse? It depend on what you mean though because China winning is also correlated with US/China being close which is also probably correlated with more racing and thus probably more misaligned AI takeover risk?)
To clarify, by “takeover” here do you mean “misaligned AI takeover”? I.e. does your “no takeover” conditional include worlds where e.g. the CCP uses AI to takeover?
Yes, I just meant “misaligned ai takeover”. Edited to clarify.
I can think of plenty of reasons, of varying levels of sensibility.
Arguments
Some people believe that (a) controlled on-paradigm ASI is possible, but that (b) it would require spending some nontrivial amount of resources/time on alignment/control research[1], and that (c) the US AGI labs are much more likely to do it than the Chinese ones. Therefore, the US winning is less likely to lead to omnicide.
I think it’s not unreasonable to believe (c), so if you believe (a) and (b), as many people do, the conclusion checks out. I assign low (but nonzero) probability to (a), though.
Even if the Chinese labs can keep ASI aligned/under control, some people are scared of being enslaved by the CCP, and think that the USG becoming god is going to be better for them.[2] This probably includes people who profess to only care about the nobody-should-build-it thing: they un/semi-consciously track the S-risk possibility, and it’s awful-feeling enough to affect their thinking even if they assign it low probability.
I think that’s a legitimate worry; S-risks are massively worse than X-risks. But I don’t expect the USG’s apotheosis to look pretty either, especially not under the current administration, and same for the apotheosis of most AGI labs, so the point is mostly moot.
I guess Anthropic or maybe DeepMind could choose non-awful results? So sure, if the current paradigm can lead to controlled ASI, and the USG stays asleep, and Anthropic/DM are the favorites to win, “make China lose” has some sense.
Variant on the above scenarios, but which does involve an international pause, with some coordination to only develop ASI once it can be kept under control. This doesn’t necessarily guarantee that the ASI, once developed, will be eudaimonic, so “who gets to ASI first/has more say on ASI” may matter; GOTO (2).
The AI-Risk advocates may feel that they have more influence on the leadership of the US labs. For US-based advocates, this is almost certainly correct. If that leadership can be convinced to pause, this buys us as much time as it’d take for the runners-up to catch up. Thus, the further behind China is, the more time we can buy in this hypothetical.
In addition, if China is way behind, it’s more likely that the US AGI labs would agree to stop, since more time to work would increase the chances of success of [whatever we want the pause for, e. g. doing alignment research or trying to cause an international ban].
Same as (4), but for governments. Perhaps the USG is easier to influence into arguing for an international pause. If so, both (a) the USG is more likely to do this if it feels that it’s comfortably ahead of China rather than nose-to-nose, (b) China is more likely to agree to an international ban if the USG is speaking from a position of power and is ahead on AI than if it’s behind/nose-to-nose. (Both because the ban would be favorable to China geopolitically, and because the X-risk arguments would sound more convincing if they don’t look like motivated reasoning/bullshit you’re inventing to convince China to abandon a technology that gives it geopolitical lead on the US.)
Some less sensible/well-thought-out variants of the above, e. g.:
Having the illusion of having more control over the US labs/government.
Semi/un-consciously feeling that it’d be better if your nation ends the world than if the Chinese do it.
Semi/un-consciously feeling that it’d be better if your nation is more powerful/ahead of a foreign one, independent of any X-risk considerations.
Suppose you think the current paradigm doesn’t scale to ASI, or that we’ll succeed in internationally banning ASI research. The amount of compute at a nation’s disposal is likely to still be increasingly important in the coming future (just because it’d allow to better harness the existing AI technology, for military and economic ends). Thus, constricting China’s access is likely to be better for the US as well.
This has nothing to do with X-risks though, it’s prosaic natsec stuff.
tl;dr:
If we get alignment by default, some US-based actors winning may be more likely to lead to a good future than the Chinese actors winning.
If on-paradigm ASI alignment is possible given some low-but-nontrivial resource expenditure, the US labs may be more likely to spend the resources on it than the Chinese ones.
US AI Safety advocates may have more control over the US AGI labs and/or the USG. The more powerful those are relative to the foreign AGI researchers, the more leverage that influence provides, including for slowing down/banning AGI research.
US AI Safety advocates may be at least partly motivated by dumb instincts for “my nation good, their nation bad”, and therefore want the US to win even if it’s winning a race-to-suicide.
Keeping a compute lead may be geopolitically important even in non-ASI worlds.
E. g., Ryan Greenblatt thinks that spending just 5% more resources than is myopically commercially expedient would drive the risk down to 50%. AI 2027 also assumes something like this.
E. g., I think this is the position of Leopold Aschenbrenner.
TBC, my view isn’t that this is sufficient for avoiding takeover risk, it is that this suffices for “you [to] have a reasonable chance of avoiding AI takeover (maybe 50% chance of misaligned AI takeover?)”.
(You seem to understand that this is my perspective and I think this is also mostly clear from the context in the box, but I wanted to clarify this given the footnote might be read in isolation or misinterpreted.)
Edited for clarity.
I’m curious, what’s your estimate for how much resources it’d take to drive the risk down to 25%, 10%, 1%?
There was some previous discussion of this in Why Should I Assume CCP AGI is Worse Than USG AGI? See e.g., my comment there:
I don’t necessarily disagree, but one big thing is freedom of speech. If the party line is to go big on AI, which is likely, given current investments, I’m not betting on Chinese Yudkowsky. Same with frontier lab whistleblowers, etc