They are worried about competition from other nations, specifically from China
The People’s Republic of China are probably closer than the United Arab Emirates, and definitely closer than the Russians, or the North Koreans, or random Somali warlords. However, as the cost of compute comes down and algorithmic efficiency improvements continue, the number of players with the needed resources will increase (especially if significant parts of the necessary technology are open-sourced). The PRC are also arguably perhaps less likely than some of the others on that list to unintentionally kill us all (they are pretty familiar with using technological means to control the thinking of intelligent agents, after all), and thus more likely to instead actually succeed in enshrining the permanent global rule of the Chinese Communist Party into the terminal goal of a sovereign superintelligence. However, judging by their actions (as opposed to their public ideology), I think they are fairly unlikely to have an AI implement anything resembling actual CEV.
If we (the West, the G20, or the UN) decided to try to enforce an actual extended pause, then access to sufficient compute, talent, and technical know-how would need to be strictly prevented for everyone, I would imagine using the intelligence and military resources of most of the leading nations in the world. Moore’s Law, academic publishing, and the open-source community would each make this increasingly difficult. (Bear in mind that the Chinese are currently buying high-end gaming graphics cards and doing some resoldering work on them to turn them into usable AI training compute.) We would thus be building up a large overhang.
I’d really love to hear some serious, practical proposals for how we, as a civilization or international community, could implement an extended pause (preferably without requiring anything in “pivotal act” territory.)
The People’s Republic of China are probably closer than the United Arab Emirates, and definitely closer than the Russians, or the North Koreans, or random Somali warlords. However, as the cost of compute comes down and algorithmic efficiency improvements continue, the number of players with the needed resources will increase (especially if significant parts of the necessary technology are open-sourced). The PRC are also arguably perhaps less likely than some of the others on that list to unintentionally kill us all (they are pretty familiar with using technological means to control the thinking of intelligent agents, after all), and thus more likely to instead actually succeed in enshrining the permanent global rule of the Chinese Communist Party into the terminal goal of a sovereign superintelligence. However, judging by their actions (as opposed to their public ideology), I think they are fairly unlikely to have an AI implement anything resembling actual CEV.
If we (the West, the G20, or the UN) decided to try to enforce an actual extended pause, then access to sufficient compute, talent, and technical know-how would need to be strictly prevented for everyone, I would imagine using the intelligence and military resources of most of the leading nations in the world. Moore’s Law, academic publishing, and the open-source community would each make this increasingly difficult. (Bear in mind that the Chinese are currently buying high-end gaming graphics cards and doing some resoldering work on them to turn them into usable AI training compute.) We would thus be building up a large overhang.
I’d really love to hear some serious, practical proposals for how we, as a civilization or international community, could implement an extended pause (preferably without requiring anything in “pivotal act” territory.)