At the heart of this concern might be a question of whether goodness is just inherently anti-competitive. What if the coalitions that preserve human values, that invest in safety, beauty and wellbeing, and that maintain meaningful human agency, are systematically outcompeted by coalitions that don’t?
It seems to me that, unlike important problems that have been described in detail not by you,[1]coordination is at least as easy as a superintelligence-enabled strategy: assuming well-soluble superalignment, the authors of the AI-2027 forecast thought it easy to “codesign a new AI, Consensus-1, whose primary imperative—taking precedence over any future orders or retraining attempts—is to enforce the terms” of the treaty between the two ASIs created in the USA and China.
How difficult does alignment need to be for your failure mode in the form of multipolar competition annihilating human values to occur, if the aligned and misaligned coalition might simply nuke the world into oblivion?
It seems to me that, unlike important problems that have been described in detail not by you,[1] coordination is at least as easy as a superintelligence-enabled strategy: assuming well-soluble superalignment, the authors of the AI-2027 forecast thought it easy to “codesign a new AI, Consensus-1, whose primary imperative—taking precedence over any future orders or retraining attempts—is to enforce the terms” of the treaty between the two ASIs created in the USA and China.
While the scenario itself had an aligned Safer-4 and a misaligned DeepCent-2, the Rogue Replication variant had the USA and China create Safer-1 and DeepCent-1 who adviced their respective governments to codesign Consensus-1. Since it had both the agenda of the USG and the CCP in mind, the world doesn’t end up being controlled by the USA, but does end up flourishing.
How difficult does alignment need to be for your failure mode in the form of multipolar competition annihilating human values to occur, if the aligned and misaligned coalition might simply nuke the world into oblivion?
However, neither me nor Claude Sonnet 4.5 think that you accounted for the Intelligence Curse. I strongly suspect that it is the most important aspect that mankind is to consider.