Many correctly recognise that the current Chinese state (and probably also the CCP regime) is mildly more authoritarian than the current US state. (Some note that the current US regime raises questions, but please don’t confuse the regime with the state, especially in a context where regime changes are frequent, as in the US.)
Likely correctly, many also note that even contemporary, and certainly near-future AI, could be quite effective for concentrating and entrenching power, especially through scaled surveillance (mostly via automated analysis) and scaled coercion (e.g. maximally via weaponised drones and robots, more minimally through existing state policing, permissioning, credit, and so on).
Some fear that China and other authoritarian states may go down this path. Some fear that the US and other somewhat authoritarian states may go down this path. (Either triggered by existing state/regime or in a coup of some kind.) This could be bad because directly bad, and also bad because it’s probably not a good setting to encounter later technological transitions and/or singularity from, which might come soon[1].
Separately, some people think that AGI and/or superintelligence would yield a decisive strategic advantage (DSA) i.e. unassailable and uncounterable ability to project force and will (note that I think this is tremendously underexamined, and most people are utterly vibing when they talk about this). Or even short of that, just some large military advantage[2] that might give an imperialist power more confidence in grabbing territory, perhaps even world domination. If CCP does this, the argument goes, the world gets more authoritarian probably. (If another does this, does it get less?)
Taking most of the above together, and choosing parameters and credences somewhat carefully, that can give some kind of (I think sloppy!) argument that, absent coordination, racing is justified. The entirely missing mood is that this racing is tremendously regretful and we should do what we can to coordinate out of it. If you rule out DSA but still think there’s gradual economic, military, and industrial advantage (I think this is something like ‘mainstream automation-pilled perspective’ held by many govt and investors) that looks like even more reason to race.
Separately again, many people think that AGI and beyond will be very difficult to control, leading to either acute or gradual disempowerment/destruction. That tends to point people toward the frame that racing is terrible!: it’s just competing to be the one with the privilege of killing everyone (including self). (Others still think that would be good actually, for misanthropic or other reasons.) This is maybe central LW perspective?
Others think the cat is mostly out of the bag, motivating coordination for pacing, and unilateral adaptation and defence work, while aiming to get coordinated and secure enough to delay, prevent (or somehow survive?) a transition far past AGI.
There’s some case that could be made in favour of a more locked-down society in the run up to a singularity—I think some versions of the superalignment kind of strategy are implicitly imagining this, in fact; rarely spoken out lout.
My main candidate for this is plain old industrial capacity, with something like ‘better combined ISTAR’ as a close second. But your enemy can still MADly nuke you in those worlds.
This seems like a pretty good summary of various perspectives. One small thing:
- “Some note that the current US regime raises questions, but please don’t confuse the regime with the state, especially in a context where regime changes are frequent, as in the US”
Point taken, but I would draw attention to the relative continuity in US foreign policy between administrations (with notable exceptions). There’s a reason Ben Rhodes talked about hating the blob. One recent example: both Biden and Trump stood firmly behind Israel even as the global outcry got ever louder. My joke is that if you trained an ASI on US foreign policy over the last 80 years its first act would be to express deranged hatred for Cuba.
I’m curious, where do you stand/which concerns do you give the most weight to? Feel free to link to an article of yours.
Do you think that remark is where the downvotes are coming from? :L
Authoritarianism is mostly a domestic question rather than foreign so I don’t know if that has much bearing. (Though I’d note that foreign policy has also experienced what looks like some movement recently.)
The articles linked in the original comment are three of mine (plus one Drexler). I’d say I’m concerned about all of catastrophic misuse, power entrenchment, gradual disempowerment, and (at some point, probably after we face some amount of those challenges) acute disempowerment/extinction.
unless something unforeseen happens, China is not winning the AI race
the weird thing is that current US regime has been relaxing chip controls which seem like the main constraint and leverage point.
Separately, because I think industrial automation might be where most of the geostrategic wins from AI come from, I actually think China has a more reasonable shot at that than you’re giving credit to. This ‘win’ looks less decisive and permanent to me than the imagined ‘superintelligence (or its wielder) takes over the world in short order’ but it might ultimately be where balance of relative global power comes from (which might affect future use of space resources and the way that subsequent tech and societal progress plays out).
“the weird thing is that current US regime has been relaxing chip controls which seem like the main constraint and leverage point”
They have been, but as noted in the article so far China is saying “no”. The relaxation is also for H200s, so a generation (and I think soon to be two generations?) behind.
You make a good point with industrial automation. I suppose my belief is that should it achieve a certain level of AGI, the US will find the political will to industrialise quickly (it already has the wealth to do so, though there are bottlenecks in certain resources as I understand it). The broad technical know-how that’s one of China’s advantages in this area would be overcome by the AGI’s advice. You could be right though, the nature of this discussion is we’re all speculating a lot.
People are having different arguments because they weight various concerns differently (and sometimes also because they’re mind-killed).
Many correctly recognise that the current Chinese state (and probably also the CCP regime) is mildly more authoritarian than the current US state. (Some note that the current US regime raises questions, but please don’t confuse the regime with the state, especially in a context where regime changes are frequent, as in the US.)
Likely correctly, many also note that even contemporary, and certainly near-future AI, could be quite effective for concentrating and entrenching power, especially through scaled surveillance (mostly via automated analysis) and scaled coercion (e.g. maximally via weaponised drones and robots, more minimally through existing state policing, permissioning, credit, and so on).
Some fear that China and other authoritarian states may go down this path. Some fear that the US and other somewhat authoritarian states may go down this path. (Either triggered by existing state/regime or in a coup of some kind.) This could be bad because directly bad, and also bad because it’s probably not a good setting to encounter later technological transitions and/or singularity from, which might come soon [1] .
Separately, some people think that AGI and/or superintelligence would yield a decisive strategic advantage (DSA) i.e. unassailable and uncounterable ability to project force and will (note that I think this is tremendously underexamined, and most people are utterly vibing when they talk about this). Or even short of that, just some large military advantage [2] that might give an imperialist power more confidence in grabbing territory, perhaps even world domination. If CCP does this, the argument goes, the world gets more authoritarian probably. (If another does this, does it get less?)
Taking most of the above together, and choosing parameters and credences somewhat carefully, that can give some kind of (I think sloppy!) argument that, absent coordination, racing is justified. The entirely missing mood is that this racing is tremendously regretful and we should do what we can to coordinate out of it. If you rule out DSA but still think there’s gradual economic, military, and industrial advantage (I think this is something like ‘mainstream automation-pilled perspective’ held by many govt and investors) that looks like even more reason to race.
Separately again, many people think that AGI and beyond will be very difficult to control, leading to either acute or gradual disempowerment/destruction. That tends to point people toward the frame that racing is terrible!: it’s just competing to be the one with the privilege of killing everyone (including self). (Others still think that would be good actually, for misanthropic or other reasons.) This is maybe central LW perspective?
Others think the cat is mostly out of the bag, motivating coordination for pacing, and unilateral adaptation and defence work, while aiming to get coordinated and secure enough to delay, prevent (or somehow survive?) a transition far past AGI.
There’s some case that could be made in favour of a more locked-down society in the run up to a singularity—I think some versions of the superalignment kind of strategy are implicitly imagining this, in fact; rarely spoken out lout.
My main candidate for this is plain old industrial capacity, with something like ‘better combined ISTAR’ as a close second. But your enemy can still MADly nuke you in those worlds.
This seems like a pretty good summary of various perspectives. One small thing:
- “Some note that the current US regime raises questions, but please don’t confuse the regime with the state, especially in a context where regime changes are frequent, as in the US”
Point taken, but I would draw attention to the relative continuity in US foreign policy between administrations (with notable exceptions). There’s a reason Ben Rhodes talked about hating the blob. One recent example: both Biden and Trump stood firmly behind Israel even as the global outcry got ever louder. My joke is that if you trained an ASI on US foreign policy over the last 80 years its first act would be to express deranged hatred for Cuba.
I’m curious, where do you stand/which concerns do you give the most weight to? Feel free to link to an article of yours.
Do you think that remark is where the downvotes are coming from? :L
Authoritarianism is mostly a domestic question rather than foreign so I don’t know if that has much bearing. (Though I’d note that foreign policy has also experienced what looks like some movement recently.)
The articles linked in the original comment are three of mine (plus one Drexler). I’d say I’m concerned about all of catastrophic misuse, power entrenchment, gradual disempowerment, and (at some point, probably after we face some amount of those challenges) acute disempowerment/extinction.
Re
the weird thing is that current US regime has been relaxing chip controls which seem like the main constraint and leverage point.
Separately, because I think industrial automation might be where most of the geostrategic wins from AI come from, I actually think China has a more reasonable shot at that than you’re giving credit to. This ‘win’ looks less decisive and permanent to me than the imagined ‘superintelligence (or its wielder) takes over the world in short order’ but it might ultimately be where balance of relative global power comes from (which might affect future use of space resources and the way that subsequent tech and societal progress plays out).
“the weird thing is that current US regime has been relaxing chip controls which seem like the main constraint and leverage point”
They have been, but as noted in the article so far China is saying “no”. The relaxation is also for H200s, so a generation (and I think soon to be two generations?) behind.
You make a good point with industrial automation. I suppose my belief is that should it achieve a certain level of AGI, the US will find the political will to industrialise quickly (it already has the wealth to do so, though there are bottlenecks in certain resources as I understand it). The broad technical know-how that’s one of China’s advantages in this area would be overcome by the AGI’s advice. You could be right though, the nature of this discussion is we’re all speculating a lot.