Just briefly skimming this, it strikes me that bounded-concern AIs are not straightforwardly a Nash Equilibrium for roughly the same reasons that the most impactful humans in the world tend to be the most ambitious.
Trying to get reality to do something that it fundamentally doesn’t want is probably a bad strategy; some group of AIs either deliberately or via misalignment decides to be unbounded and then it has a huge advantage...
You’ve precisely identified the central crux. This dynamic — whether bounded communities can be stable against unbounded optimization — was a main focus of my recent conversation with Plex: Symbiogenesis vs. Convergent Consequentialism.
Your intuition holds true for offense-dominant environments. The d/acc move is to actively shape the environment toward defense-dominance. In such a reality, bounded, cooperative strategies (symbiogenesis) become evolutionarily stable, and the most ambitious strategy is robust cooperation, rather than unilateral vampirism.
Just briefly skimming this, it strikes me that bounded-concern AIs are not straightforwardly a Nash Equilibrium for roughly the same reasons that the most impactful humans in the world tend to be the most ambitious.
Trying to get reality to do something that it fundamentally doesn’t want is probably a bad strategy; some group of AIs either deliberately or via misalignment decides to be unbounded and then it has a huge advantage...
You’ve precisely identified the central crux. This dynamic — whether bounded communities can be stable against unbounded optimization — was a main focus of my recent conversation with Plex: Symbiogenesis vs. Convergent Consequentialism.
Your intuition holds true for offense-dominant environments. The d/acc move is to actively shape the environment toward defense-dominance. In such a reality, bounded, cooperative strategies (symbiogenesis) become evolutionarily stable, and the most ambitious strategy is robust cooperation, rather than unilateral vampirism.