I think the point is that there aren’t just two black and white possibilities:
get extinct by developing superintelligence,
stop AI R&D.
I would argue that it is much more difficult to understand at which point the AI is too dangerous to be improved than to understand at which point YOUR AI is “better enough” than YOUR ENEMY’S AI (China or the US) to provide you some strategic advantage. Under this premise, if your adversary is rational, once you stop your AI R&D, he may realize that his optimal choice is to do just a bit more R&D and only then stop, in order to get some strategic advantages out of it, and still avoid the catastrophic risk of extinction.
If both adversaries consider the other two possibilities:
develop “just a bit more AI” to catch up with the other country,
develop “just a bit more AI” to get some strategic advantage over the other country...
they can push the Nash equilibrium into “get extinct by developing superintelligence”.
On the other hand, I’d say that in order to hope for some international cooperation you need to have some more nuance (maybe GPT-5.5 level) and model the fact that possibly both good and bad guys in their heart believe to be the good guys, don’t want to see the world burn, and may be happy to take a leap of faith and stop/regulate AI R&D in the same way as we did for weapons/pollution: not in pursuit of their selfish interest, but because it is good for the world.
I think the point is that there aren’t just two black and white possibilities:
get extinct by developing superintelligence,
stop AI R&D.
I would argue that it is much more difficult to understand at which point the AI is too dangerous to be improved than to understand at which point YOUR AI is “better enough” than YOUR ENEMY’S AI (China or the US) to provide you some strategic advantage. Under this premise, if your adversary is rational, once you stop your AI R&D, he may realize that his optimal choice is to do just a bit more R&D and only then stop, in order to get some strategic advantages out of it, and still avoid the catastrophic risk of extinction.
If both adversaries consider the other two possibilities:
develop “just a bit more AI” to catch up with the other country,
develop “just a bit more AI” to get some strategic advantage over the other country...
they can push the Nash equilibrium into “get extinct by developing superintelligence”.
On the other hand, I’d say that in order to hope for some international cooperation you need to have some more nuance (maybe GPT-5.5 level) and model the fact that possibly both good and bad guys in their heart believe to be the good guys, don’t want to see the world burn, and may be happy to take a leap of faith and stop/regulate AI R&D in the same way as we did for weapons/pollution: not in pursuit of their selfish interest, but because it is good for the world.