I have broad agreement with this overall document, with some relatively minor/subjective disagreements on what would be the optimal point to pause further capabilities work[1], but unless A) I have missed the section where you address this directly, or B) you have deliberately omitted this for strategic reasons, there does seem to be a serious oversight in the current plan that could render it unviable unless a solution to it is found:
You correctly point out that it is in the interests of China to agree to this treaty, but have not explained why it is in the interests of the CCP, Xi, and Xi’s inner circle to go along with a treaty which would make the average Chinese citizen much wealthier than they are today, but would greatly reduce the hard and soft power of those specific Chinese elites in both absolute and relative (to their own citizens) terms. Instead, what AI 2040 seemingly does is lump these groups together into the general concept handle of “China”, and then conflate what would be good for China as a whole with what is in the self-interest of the specific Chinese elites (and their various internal power struggles) who will actually make the decision on whether to sign such a treaty.
This oversight on the Chinese-side is not something that you made whatsoever when talking about the US-side, where fine distinctions are made between the interests and motives of the President, Congress and the heads of multiple frontier AI labs, with each of their actions modelled accordingly. (And yeah, I do appreciate that acquiring the kind of information needed to build a similarly detailed model of what’s going on in China is extremely hard as an outsider, but unfortunately that does not mean this problem can be hand-waved away.)
Your approximate position being “We should pause AI just before we lose control entirely, at the level where it can provide a $1m/year UBI and 70% of people in America & First World countries have been made permanently unemployable, and this will be pretty good overall despite all of the novel social problems it will create, because $1m GDP/capita allows us to basically buy our way out of them.” vs my approximate position of “We should pause general capability work ASAP, allow a few narrow use cases to continue like drug development to cure all curable diseases, and just let the general capabilities of 2026 that we’ve barely scratched the surface on diffuse through the economy like a second and even bigger computer revolution over the next 50-100 years, alongside the general productivity improvements that come from non-AI fields.” with the implicit logic behind my position being “even if it wasn’t crazy to leave so little margin of safety to the loss-of-control threshold, most people don’t actually want to live in any version of this AI-enabled transhumanist utopia that would actually exist, and even the $1m/person/year of goods and services that could be bought in a nominally post-scarcity economy will not be able to compensate for the new problems it would create and the existing problems it would magnify”.
I have broad agreement with this overall document, with some relatively minor/subjective disagreements on what would be the optimal point to pause further capabilities work [1] , but unless A) I have missed the section where you address this directly, or B) you have deliberately omitted this for strategic reasons, there does seem to be a serious oversight in the current plan that could render it unviable unless a solution to it is found:
You correctly point out that it is in the interests of China to agree to this treaty, but have not explained why it is in the interests of the CCP, Xi, and Xi’s inner circle to go along with a treaty which would make the average Chinese citizen much wealthier than they are today, but would greatly reduce the hard and soft power of those specific Chinese elites in both absolute and relative (to their own citizens) terms. Instead, what AI 2040 seemingly does is lump these groups together into the general concept handle of “China”, and then conflate what would be good for China as a whole with what is in the self-interest of the specific Chinese elites (and their various internal power struggles) who will actually make the decision on whether to sign such a treaty.
This oversight on the Chinese-side is not something that you made whatsoever when talking about the US-side, where fine distinctions are made between the interests and motives of the President, Congress and the heads of multiple frontier AI labs, with each of their actions modelled accordingly. (And yeah, I do appreciate that acquiring the kind of information needed to build a similarly detailed model of what’s going on in China is extremely hard as an outsider, but unfortunately that does not mean this problem can be hand-waved away.)
Your approximate position being “We should pause AI just before we lose control entirely, at the level where it can provide a $1m/year UBI and 70% of people in America & First World countries have been made permanently unemployable, and this will be pretty good overall despite all of the novel social problems it will create, because $1m GDP/capita allows us to basically buy our way out of them.” vs my approximate position of “We should pause general capability work ASAP, allow a few narrow use cases to continue like drug development to cure all curable diseases, and just let the general capabilities of 2026 that we’ve barely scratched the surface on diffuse through the economy like a second and even bigger computer revolution over the next 50-100 years, alongside the general productivity improvements that come from non-AI fields.” with the implicit logic behind my position being “even if it wasn’t crazy to leave so little margin of safety to the loss-of-control threshold, most people don’t actually want to live in any version of this AI-enabled transhumanist utopia that would actually exist, and even the $1m/person/year of goods and services that could be bought in a nominally post-scarcity economy will not be able to compensate for the new problems it would create and the existing problems it would magnify”.