I have broad agreement with this overall document, with some relatively minor/subjective disagreements on what would be the optimal point to pause further capabilities work[1], but unless A) I have missed the section where you address this directly, or B) you have deliberately omitted this for strategic reasons, there does seem to be a serious oversight in the current plan that could render it unviable unless a solution to it is found:
You correctly point out that it is in the interests of China to agree to this treaty, but have not explained why it is in the interests of the CCP, Xi, and Xi’s inner circle to go along with a treaty which would make the average Chinese citizen much wealthier than they are today, but would greatly reduce the hard and soft power of those specific Chinese elites in both absolute and relative (to their own citizens) terms. Instead, what AI 2040 seemingly does is lump these groups together into the general concept handle of “China”, and then conflate what would be good for China as a whole with what is in the self-interest of the specific Chinese elites (and their various internal power struggles) who will actually make the decision on whether to sign such a treaty.
This oversight on the Chinese-side is not something that you made whatsoever when talking about the US-side, where fine distinctions are made between the interests and motives of the President, Congress and the heads of multiple frontier AI labs, with each of their actions modelled accordingly. (And yeah, I do appreciate that acquiring the kind of information needed to build a similarly detailed model of what’s going on in China is extremely hard as an outsider, but unfortunately that does not mean this problem can be hand-waved away.)
Your approximate position being “We should pause AI just before we lose control entirely, at the level where it can provide a $1m/year UBI and 70% of people in America & First World countries have been made permanently unemployable, and this will be pretty good overall despite all of the novel social problems it will create, because $1m GDP/capita allows us to basically buy our way out of them.” vs my approximate position of “We should pause general capability work ASAP, allow a few narrow use cases to continue like drug development to cure all curable diseases, and just let the general capabilities of 2026 that we’ve barely scratched the surface on diffuse through the economy like a second and even bigger computer revolution over the next 50-100 years, alongside the general productivity improvements that come from non-AI fields.” with the implicit logic behind my position being “even if it wasn’t crazy to leave so little margin of safety to the loss-of-control threshold, most people don’t actually want to live in any version of this AI-enabled transhumanist utopia that would actually exist, and even the $1m/person/year of goods and services that could be bought in a nominally post-scarcity economy will not be able to compensate for the new problems it would create and the existing problems it would magnify”.
The Chinese leaders’ alternative is to lose the ASI race to the US, which puts them in significantly higher danger of being disempowered or even killed. And given the trajectory we’re on, they would need major algorithmic breakthroughs, that they successfully manage to keep secret, to not be in a position where they’re going to lose. They’re at too large of a compute disadvantage and the timelines are too short for them to use their better industrial capacity to overcome that. That might be more uncertain if it was clear that they were already treating the situation with the appropriate gravity, but they are not. They seem to be viewing AI the same way the US gov currently does, as a mere important economic and military technology.
I have broad agreement with this overall document, with some relatively minor/subjective disagreements on what would be the optimal point to pause further capabilities work [1] , but unless A) I have missed the section where you address this directly, or B) you have deliberately omitted this for strategic reasons, there does seem to be a serious oversight in the current plan that could render it unviable unless a solution to it is found:
You correctly point out that it is in the interests of China to agree to this treaty, but have not explained why it is in the interests of the CCP, Xi, and Xi’s inner circle to go along with a treaty which would make the average Chinese citizen much wealthier than they are today, but would greatly reduce the hard and soft power of those specific Chinese elites in both absolute and relative (to their own citizens) terms. Instead, what AI 2040 seemingly does is lump these groups together into the general concept handle of “China”, and then conflate what would be good for China as a whole with what is in the self-interest of the specific Chinese elites (and their various internal power struggles) who will actually make the decision on whether to sign such a treaty.
This oversight on the Chinese-side is not something that you made whatsoever when talking about the US-side, where fine distinctions are made between the interests and motives of the President, Congress and the heads of multiple frontier AI labs, with each of their actions modelled accordingly. (And yeah, I do appreciate that acquiring the kind of information needed to build a similarly detailed model of what’s going on in China is extremely hard as an outsider, but unfortunately that does not mean this problem can be hand-waved away.)
Your approximate position being “We should pause AI just before we lose control entirely, at the level where it can provide a $1m/year UBI and 70% of people in America & First World countries have been made permanently unemployable, and this will be pretty good overall despite all of the novel social problems it will create, because $1m GDP/capita allows us to basically buy our way out of them.” vs my approximate position of “We should pause general capability work ASAP, allow a few narrow use cases to continue like drug development to cure all curable diseases, and just let the general capabilities of 2026 that we’ve barely scratched the surface on diffuse through the economy like a second and even bigger computer revolution over the next 50-100 years, alongside the general productivity improvements that come from non-AI fields.” with the implicit logic behind my position being “even if it wasn’t crazy to leave so little margin of safety to the loss-of-control threshold, most people don’t actually want to live in any version of this AI-enabled transhumanist utopia that would actually exist, and even the $1m/person/year of goods and services that could be bought in a nominally post-scarcity economy will not be able to compensate for the new problems it would create and the existing problems it would magnify”.
The Chinese leaders’ alternative is to lose the ASI race to the US, which puts them in significantly higher danger of being disempowered or even killed. And given the trajectory we’re on, they would need major algorithmic breakthroughs, that they successfully manage to keep secret, to not be in a position where they’re going to lose. They’re at too large of a compute disadvantage and the timelines are too short for them to use their better industrial capacity to overcome that. That might be more uncertain if it was clear that they were already treating the situation with the appropriate gravity, but they are not. They seem to be viewing AI the same way the US gov currently does, as a mere important economic and military technology.