I’m suspect that when things have come to enough of a head that AI companies are thinking of pulling the internal alarm bell and doing a hard switch to safety, that there’s going to be a significant amount of interest from the government.
I anticipate a high likelihood that sufficiently powerful AI will represent a seriously destabilizing force on international power and relations. I think that this will probably lead to the governments whose countries the labs are based in deciding to nationalize all the labs and declare it illegal for AI researchers to emigrate or work for anyone but the government. I also suspect that the government will not agree to pause, even if the AI companies would have if left to their own devices. I believe the unavoidable military impact will pressure the government into going full-speed-ahead on military applications of AI. If there are some governments that do this, but some that don’t, I expect that the world will quickly become militarily dominated by the governments that do. This could potentially be one government, like the US before anyone else had nuclear weapons.
Basically, I anticipate that sufficiently powerful AI will unlock technology able to guarantee an enemy country will not be able to launch any land or sea based missiles or air attacks. That’d be a big deal. If the US government found out that OpenAI had discovered such a technology, how do you think the US military would react? They certainly wouldn’t want to chance the tech leaking to Russia or China and thus unempowering the USA & allies.
I agree with the broader claim that as AGI approaches, governments are likely to intervene drastically to deal with national security threats.
However, I’m not so sure about the “therefore a global arms race will start” claim. I think it’s pretty plausible that if the US or UK are the first to approach AGI, that they would come to their senses and institute a global pause instead of spearheading an arms race. Although maybe that’s wishful thinking on my part.
I expect some people in the government to be like “wait, if a global arms race starts this is likely to end in catastrophe” and advocate for a pause instead. I think the US would be pretty happy with an enforcable pause if this meant it got to maintain a slight lead. I’d hope that (pause+slight lead) would be much more inticing than (race+large lead) given the catastrophic risk associated with the latter.
I agree actually. I think the three most likely branches of the future are:
a) a strong and well-enforced international treaty between all major powers which no country can opt out of, that successfully controls AI development and deployment.
b) a race-to-the-bottom of arms races, and first strikes, and increasing relative breakdowns in state monopoly on force as AI grants novel uncontrolled weapons tech to smaller groups (e.g. terrorist orgs).
c) (least likely, but still possible) a sudden jump in power of AI, which allows the controlling entity to seize power of the entire world in such a short time that state actors cannot react or stop them.
I’m suspect that when things have come to enough of a head that AI companies are thinking of pulling the internal alarm bell and doing a hard switch to safety, that there’s going to be a significant amount of interest from the government.
I anticipate a high likelihood that sufficiently powerful AI will represent a seriously destabilizing force on international power and relations. I think that this will probably lead to the governments whose countries the labs are based in deciding to nationalize all the labs and declare it illegal for AI researchers to emigrate or work for anyone but the government. I also suspect that the government will not agree to pause, even if the AI companies would have if left to their own devices. I believe the unavoidable military impact will pressure the government into going full-speed-ahead on military applications of AI. If there are some governments that do this, but some that don’t, I expect that the world will quickly become militarily dominated by the governments that do. This could potentially be one government, like the US before anyone else had nuclear weapons.
Basically, I anticipate that sufficiently powerful AI will unlock technology able to guarantee an enemy country will not be able to launch any land or sea based missiles or air attacks. That’d be a big deal. If the US government found out that OpenAI had discovered such a technology, how do you think the US military would react? They certainly wouldn’t want to chance the tech leaking to Russia or China and thus unempowering the USA & allies.
I agree with the broader claim that as AGI approaches, governments are likely to intervene drastically to deal with national security threats.
However, I’m not so sure about the “therefore a global arms race will start” claim. I think it’s pretty plausible that if the US or UK are the first to approach AGI, that they would come to their senses and institute a global pause instead of spearheading an arms race. Although maybe that’s wishful thinking on my part.
I expect some people in the government to be like “wait, if a global arms race starts this is likely to end in catastrophe” and advocate for a pause instead. I think the US would be pretty happy with an enforcable pause if this meant it got to maintain a slight lead. I’d hope that (pause+slight lead) would be much more inticing than (race+large lead) given the catastrophic risk associated with the latter.
I agree actually. I think the three most likely branches of the future are:
a) a strong and well-enforced international treaty between all major powers which no country can opt out of, that successfully controls AI development and deployment.
b) a race-to-the-bottom of arms races, and first strikes, and increasing relative breakdowns in state monopoly on force as AI grants novel uncontrolled weapons tech to smaller groups (e.g. terrorist orgs).
c) (least likely, but still possible) a sudden jump in power of AI, which allows the controlling entity to seize power of the entire world in such a short time that state actors cannot react or stop them.