I think this dynamic is true at different scales, not just humanity’s overall civilization.
The fundamental problem is that everyone’s locked in a prisoner’s dilemma with Darwinian evolution tacked on top so that those who win one round get to duplicate and gain an advantage in the next round, so that everyone has to constantly defect to gain power. (This applies even to actors who want to optimize for cooperation in the world—their best strategy is ruthlessly gaining power first to gain the ability to use coercive strategies that force other people to cooperate! Note: in the real world these actors may have a way to avoid “defecting”, or doing non-cooperative actions, if they can creatively find a way around most competiton and thus avoid the prisoner’s dilemma entirely)
As a result you can see examples throughout history of organizations and societies failing to “stop Cthulhu” (as defined by this article) in various different ways. You can replace Cthulhu with climate change, Hitler, or the long-term innovation of companies that are required to stay relevant in a changing market 20 years down the line. People never start cooperating until they realize it’s almost too late to stop the issue, and thus that banding together and focusing efforts to solve the issue becomes the only possible good strategy for all actors. And sometimes they realize too late and the society or company never recovers. (On a global civilizational level, this kind of last-minute recovery gets harder and harder as technology improves. How WW2 played out would be almost certainly impossible to replicate if something like it were to happen again. But perhaps that would just push the “last-minute” threshold earlier instead so the world keeps being saved just before the problem gets out of control.)
The only way around these dynamics is for a leader or regulator to punish defectors and force everyone to cooperate towards solving the long-term problem, who is themselves benevolent (wants to solve said long-term problem) and beyond the ability of anyone else to compete against. Examples: visionary founders, shareholder-proof leaders of PBCs, national governments (relative to companies), multinational organizations like the EU and UN (to the extent they actually do something), and the US-led post-war world order more generally.
Otherwise the short-term optimizers will always bubble to the top and doom the society or organization as a whole.
The author says as much:
This is, in effect, asking for a global coordination mechanism that persists indefinitely against strong incentives to defect.
At corporate-sized organizational scales, there are obviously feasible solutions to the problem. In the civilization case, it is almost impossible without domination by a single leader, whether human or AI.
(I would say that this line of thinking, if extrapolated to a societal and global level, leads to some very troubling implications on what the best path of future civilization looks like.)
You provided many real-world illustrations of the idea, which are indeed relevant. I did not focus in the post itself on examples or on how the ideas manifest in history or economics, but there are many local examples.
Actually, one particular case that is very close to me as a Ukrainian is the Russo-Ukrainian war. I have been constantly thinking about this case while working on the idea. Ukraine, fortunately, survived, but at a great cost, and it would not have survived at its level of preparedness if Russia had not been so extremely incompetent. Pro-survival agents were largely outcompeted in Ukraine by survival-indifferent agents, and a competent adversary motivated to exterminate Ukraine would have succeeded rather easily. Later, because the war persisted for a long time and provided a tight feedback loop, society reorganized itself around being pro-survival, but that happened only because the initial Russian invasion failed. The pressure from the existential threat was slow enough to allow adjustment, and in a way it was a great stroke of luck that it was slow enough.
I think this dynamic is true at different scales, not just humanity’s overall civilization.
The fundamental problem is that everyone’s locked in a prisoner’s dilemma with Darwinian evolution tacked on top so that those who win one round get to duplicate and gain an advantage in the next round, so that everyone has to constantly defect to gain power. (This applies even to actors who want to optimize for cooperation in the world—their best strategy is ruthlessly gaining power first to gain the ability to use coercive strategies that force other people to cooperate! Note: in the real world these actors may have a way to avoid “defecting”, or doing non-cooperative actions, if they can creatively find a way around most competiton and thus avoid the prisoner’s dilemma entirely)
As a result you can see examples throughout history of organizations and societies failing to “stop Cthulhu” (as defined by this article) in various different ways. You can replace Cthulhu with climate change, Hitler, or the long-term innovation of companies that are required to stay relevant in a changing market 20 years down the line. People never start cooperating until they realize it’s almost too late to stop the issue, and thus that banding together and focusing efforts to solve the issue becomes the only possible good strategy for all actors. And sometimes they realize too late and the society or company never recovers. (On a global civilizational level, this kind of last-minute recovery gets harder and harder as technology improves. How WW2 played out would be almost certainly impossible to replicate if something like it were to happen again. But perhaps that would just push the “last-minute” threshold earlier instead so the world keeps being saved just before the problem gets out of control.)
The only way around these dynamics is for a leader or regulator to punish defectors and force everyone to cooperate towards solving the long-term problem, who is themselves benevolent (wants to solve said long-term problem) and beyond the ability of anyone else to compete against. Examples: visionary founders, shareholder-proof leaders of PBCs, national governments (relative to companies), multinational organizations like the EU and UN (to the extent they actually do something), and the US-led post-war world order more generally.
Otherwise the short-term optimizers will always bubble to the top and doom the society or organization as a whole.
The author says as much:
At corporate-sized organizational scales, there are obviously feasible solutions to the problem. In the civilization case, it is almost impossible without domination by a single leader, whether human or AI.
(I would say that this line of thinking, if extrapolated to a societal and global level, leads to some very troubling implications on what the best path of future civilization looks like.)
You provided many real-world illustrations of the idea, which are indeed relevant. I did not focus in the post itself on examples or on how the ideas manifest in history or economics, but there are many local examples.
Actually, one particular case that is very close to me as a Ukrainian is the Russo-Ukrainian war. I have been constantly thinking about this case while working on the idea. Ukraine, fortunately, survived, but at a great cost, and it would not have survived at its level of preparedness if Russia had not been so extremely incompetent. Pro-survival agents were largely outcompeted in Ukraine by survival-indifferent agents, and a competent adversary motivated to exterminate Ukraine would have succeeded rather easily. Later, because the war persisted for a long time and provided a tight feedback loop, society reorganized itself around being pro-survival, but that happened only because the initial Russian invasion failed. The pressure from the existential threat was slow enough to allow adjustment, and in a way it was a great stroke of luck that it was slow enough.