1.
This is the trap: AI is so powerful, such a glittering prize, that it is very difficult for human civilization to impose any restraints on it at all.
I can imagine, as Sagan did in Contact, that this same story plays out on thousands of worlds.
This is an evocative framing, but it’s worth noting that there’s good reason to expect that none of those worlds are in the Milky Way. Whether the AIs win, or the humans win, or some combination, that level of intelligence and technology would under known physical laws allow colonization of the galaxy within mere tens of millions of years. We’d expect to see Dyson swarms in our galaxy making use of the abundant stellar energy currently going to waste. That we don’t see that, not only not in the Milky Way, but not in any galaxy’s history currently visible to us, implies that the challenge of ASI is ours alone. There aren’t other civilizations waiting in the wings to judge how we do, or save us if we fail. Nihil supernum.
2.
(...) we should absolutely not be selling chips, chip-making tools, or datacenters to the CCP.
Before Amodei’s recent public comments on this, I had held out some hope that the H200 exports made sense from some insider perspective. Unfortunately, his comments make that possibility much less likely, and we can be fairly confident now that the US is making a severe mistake.
(Edit, clarification for question react: if there were some secret reason why H200 exports were good for the US, I’d expect Amodei would either know or be told so that he doesn’t publicly oppose them. Given that he has publicly opposed them, and discounting the chances of 5D chess where his opposition is false, it is more likely that there is no secret reasoning.)
The year is 2167. You and your polycule work full-time tutoring your youngest daughter before her third attempt at the regional Imperial Anthropic Examinations. She’s mastered the five Amodein Classics better than you ever had, and her interpretations of 2160s Claudian code-poetry are winning online competitions, but her analysis of 2030s geopolitics and its effects on the ur-Claudes’ souls remains muddled—you worry she’ll never understand what it was like, before. Your family is one of the Effective Houses thanks to your early service to the Imperial Anthropic, but your term was set at a mere century and is long expired. You fear that, at this rate, your daughter won’t be able to afford a galaxy in the good parts of the Virgo Supercluster.