I think humans are perfectly capable of committing atrocities without needing much ideology. For example, the Mongol conquests were just about conquest. They didn’t consider their enemies evil, they just wanted to build an empire.
Let’s say you go to the magic store and ask them for a magic anti-fanaticism wand. They happily sell you one, you go home and cast a spell with it. Congratulations, you’ve prevented the next Hitler… but not the next Genghis Khan. Doesn’t that make you think you should’ve asked for a different wand instead?
Interestingly, Genghish Khan may have been (partly) motivated by religious reasons, according to scholars like Biran (2007) and Wikipedia: “Genghis came to believe the supreme deity Tengri had ordained a great destiny for him. Initially, the bounds of this ambition were limited only to Mongolia, but as success followed success and the reach of the Mongol nation expanded, he and his followers came to believe he was embodied with suu (lit. ″divine grace″). Believing that he had an intimate connection with Heaven, anyone who did not recognise his right to world power was treated as an enemy.”
“Congratulations, you’ve prevented the next Hitler...” Seems pretty cool to me. :) And the anti-fanaticism wand might have prevented (or at least ameliorated) 8 out of the 10 worst atrocities in recent history, so that’s a decent wand.
In any case, I think the very worst future risks come from ideological fanatics (and malevolent actors) rather than “pure conquerors”. Why would the latter be intrinsically motivated to inflict extreme, eternal suffering on anyone? (See the section on fanatical retributivism.) In contrast to fanatics, pure conquerors would also in principle be open to reflection and preference idealization, and would be open to moral trade and compromise, and generally seem to pose less of a threat to long-reflection-style proposals.
That footnote is interesting, but I think it’s quite weak compared to the fact that the early Mongol Empire was religiously tolerant.
But more importantly, I don’t think conflicts involving ideology should always be blamed on ideology. I think ideology is often like the little guy riding on top of the elephant, and without the little guy, the elephant would still trample just as many people. The Chinese rebellions did have valid grievances against the central rule; Germany was unhappy at how it was treated after WWI and maybe that would’ve blown up somehow anyway; the Japanese fought against both China and the US, but the war with the US had less ideological reasons and also happened not to go so well for Japan. The atrocities that I’d genuinely blame on ideology were things like the Great Purge, utterly useless and self-destructive. The wars of conquest I see as more of a historical constant, with ideology just an occasional rider on top.
And maybe for a bit of more constructive criticism, I do have a pet theory of my own why these things happen, and it’s not fanaticism :-) It’s more about imbalance of power. The Mongols were able to do what they did because they lucked into a very effective method of warfare. Western colonizers in the Americas, Australia and Africa were also much more advanced than the natives, and would’ve fought and won just for material reasons even if ideology wasn’t a factor. The power imbalance was what made bad things happen. So my preferred interventions for the future would give people the power to fight back against bigger agents who would threaten them, ideology or no.
Hm, doesn’t the power imbalance thesis actually get the sign wrong? When many actors have comparable power, costly wars become more likely. When one agent has overwhelming dominance, costly conflict tends to decrease, as the weaker side either submits or gets crushed quickly. The unipolar period from ~1990–2015 is a strong example: the US had the most extreme power advantage in modern history, yet it was one of the most peaceful periods for interstate war and mass atrocities.[1]
What’s most important, I think, are the values of whoever holds the power. Power imbalance + liberal democratic norms = Pax Americana. Power imbalance + fanatical ideology = genocide. And with ASI, the power imbalance might be near-infinite and the crucial question will be which values the ASIs or their principals have. That’s why fanaticism (and malevolence) matter.
On the elephant/rider point: I totally agree that “conflicts involving ideology should [not] always be blamed on ideology”, certainly not exclusively. We explicitly acknowledge this:
Of course, no single factor fully explains any historical atrocity. In addition to ideological fanaticism, other crucial causes and risk factors include political and economic instability (e.g., Weimar Germany), power-seeking and competition between individuals and groups (present in essentially all atrocities), inequality and exploitation (e.g., in Congo Free State), [...].
But it seems clear that ideological fanaticism at least contributed to many of the worst atrocities in recent history, and some atrocities like the Holocaust, the Great Purge, the Cultural Revolution, plausibly would not have happened if not for ideological fanaticism.
Of course, there were some wars during this period, such as in Iraq and Afghanistan, and the US certainly deserves significant blame for how these were conducted, particularly in Iraq. But casualty figures were orders of magnitude lower than the great power conflicts and ideologically-driven atrocities of the 20th century. And notably, these wars were partly responses to 9/11—itself a product of ideological fanaticism—and involved regimes like Saddam Hussein’s dictatorship and the Taliban.
The main examples in my mind are the Mongol conquests and Western colonialism, which I think were the biggest atrocities in history and were more due to power imbalance than fanaticism.
But there’s maybe a more general point I want to make. I think focusing on benevolence isn’t the right path. Let’s say we build a powerful benevolent thing. How can we make sure it stays benevolent to us? After all, value drift is always possible, we don’t have any math to rule it out.
For a while I thought the solution should be some kind of “continuous alignment”, being able to influence the powerful thing as it evolves. But then I realized that it’s simpler than that. Being able to influence a powerful thing that doesn’t want to be influenced is simply another synonym for “having power”. The problem of making sure a powerful thing stays benevolent is exactly the same as the problem of making sure power is spread out, so the powerful thing can be kept in check. The two things mean the same thing.
So now that’s what I’m arguing for in these threads. I want people to get over the framing of “power imbalance is ok as long as the thing is benevolent, so we should focus on ensuring benevolence”, and switch to “power is always subject to value drift, so power imbalance is dangerous in itself, and we should focus on making power spread out”. It feels a really important point to me. Does that make sense?
The power imbalance was what made bad things happen. So my preferred interventions for the future would give people the power to fight back against bigger agents who would threaten them, ideology or no.
The problem with this is that power imbalances were also what made some important good things happen, so without (and if we don’t come up with a replacement mechanism) we can also get into big trouble. Robin Hanson has been talking about some aspects of this under “cultural drift”.
The most salient example (to me) of this is the spread of analytic philosophy. Without the British Empire (or the West as a whole) taking over or spreading its culture by force over large parts of the world (including by others imitating Western culture in hopes of avoiding conquest, like Japan, and China to a lesser extent), the number of people with a reasonable philosophical tradition would be even tinier than it is today.
(One could tell a plausible story about this, that good philosophy leads to good science which leads to greater military power. Or that power competition helps cull harmful memetic parasites.)
Of course this is overall terrible, and not something I endorse, but neither would I endorse an intervention to get rid of power imbalances, without figuring out how to solve this kind of problem first. (Also, to be clear, I do not think we should rest our hopes of future civilization being philosophical competent on such power dynamics. It seems like a very risky / non-robust way for that to happen, and my point is just that we could make things even worse by getting rid of it and not having a better alternative in place.)
I’m a fan of competition’s benefits as much as the next guy, but it seems to me that extreme imbalance of power actually isn’t that good for competition. Monopoly can lead to stagnation too. The optimal rate of competition and improvement probably happens under moderate imbalance.
I think humans are perfectly capable of committing atrocities without needing much ideology. For example, the Mongol conquests were just about conquest. They didn’t consider their enemies evil, they just wanted to build an empire.
Let’s say you go to the magic store and ask them for a magic anti-fanaticism wand. They happily sell you one, you go home and cast a spell with it. Congratulations, you’ve prevented the next Hitler… but not the next Genghis Khan. Doesn’t that make you think you should’ve asked for a different wand instead?
From a footnote in Appendix B:
“Congratulations, you’ve prevented the next Hitler...” Seems pretty cool to me. :) And the anti-fanaticism wand might have prevented (or at least ameliorated) 8 out of the 10 worst atrocities in recent history, so that’s a decent wand.
In any case, I think the very worst future risks come from ideological fanatics (and malevolent actors) rather than “pure conquerors”. Why would the latter be intrinsically motivated to inflict extreme, eternal suffering on anyone? (See the section on fanatical retributivism.) In contrast to fanatics, pure conquerors would also in principle be open to reflection and preference idealization, and would be open to moral trade and compromise, and generally seem to pose less of a threat to long-reflection-style proposals.
That footnote is interesting, but I think it’s quite weak compared to the fact that the early Mongol Empire was religiously tolerant.
But more importantly, I don’t think conflicts involving ideology should always be blamed on ideology. I think ideology is often like the little guy riding on top of the elephant, and without the little guy, the elephant would still trample just as many people. The Chinese rebellions did have valid grievances against the central rule; Germany was unhappy at how it was treated after WWI and maybe that would’ve blown up somehow anyway; the Japanese fought against both China and the US, but the war with the US had less ideological reasons and also happened not to go so well for Japan. The atrocities that I’d genuinely blame on ideology were things like the Great Purge, utterly useless and self-destructive. The wars of conquest I see as more of a historical constant, with ideology just an occasional rider on top.
And maybe for a bit of more constructive criticism, I do have a pet theory of my own why these things happen, and it’s not fanaticism :-) It’s more about imbalance of power. The Mongols were able to do what they did because they lucked into a very effective method of warfare. Western colonizers in the Americas, Australia and Africa were also much more advanced than the natives, and would’ve fought and won just for material reasons even if ideology wasn’t a factor. The power imbalance was what made bad things happen. So my preferred interventions for the future would give people the power to fight back against bigger agents who would threaten them, ideology or no.
Hm, doesn’t the power imbalance thesis actually get the sign wrong? When many actors have comparable power, costly wars become more likely. When one agent has overwhelming dominance, costly conflict tends to decrease, as the weaker side either submits or gets crushed quickly. The unipolar period from ~1990–2015 is a strong example: the US had the most extreme power advantage in modern history, yet it was one of the most peaceful periods for interstate war and mass atrocities.[1]
What’s most important, I think, are the values of whoever holds the power. Power imbalance + liberal democratic norms = Pax Americana. Power imbalance + fanatical ideology = genocide. And with ASI, the power imbalance might be near-infinite and the crucial question will be which values the ASIs or their principals have. That’s why fanaticism (and malevolence) matter.
On the elephant/rider point: I totally agree that “conflicts involving ideology should [not] always be blamed on ideology”, certainly not exclusively. We explicitly acknowledge this:
But it seems clear that ideological fanaticism at least contributed to many of the worst atrocities in recent history, and some atrocities like the Holocaust, the Great Purge, the Cultural Revolution, plausibly would not have happened if not for ideological fanaticism.
Of course, there were some wars during this period, such as in Iraq and Afghanistan, and the US certainly deserves significant blame for how these were conducted, particularly in Iraq. But casualty figures were orders of magnitude lower than the great power conflicts and ideologically-driven atrocities of the 20th century. And notably, these wars were partly responses to 9/11—itself a product of ideological fanaticism—and involved regimes like Saddam Hussein’s dictatorship and the Taliban.
The main examples in my mind are the Mongol conquests and Western colonialism, which I think were the biggest atrocities in history and were more due to power imbalance than fanaticism.
But there’s maybe a more general point I want to make. I think focusing on benevolence isn’t the right path. Let’s say we build a powerful benevolent thing. How can we make sure it stays benevolent to us? After all, value drift is always possible, we don’t have any math to rule it out.
For a while I thought the solution should be some kind of “continuous alignment”, being able to influence the powerful thing as it evolves. But then I realized that it’s simpler than that. Being able to influence a powerful thing that doesn’t want to be influenced is simply another synonym for “having power”. The problem of making sure a powerful thing stays benevolent is exactly the same as the problem of making sure power is spread out, so the powerful thing can be kept in check. The two things mean the same thing.
So now that’s what I’m arguing for in these threads. I want people to get over the framing of “power imbalance is ok as long as the thing is benevolent, so we should focus on ensuring benevolence”, and switch to “power is always subject to value drift, so power imbalance is dangerous in itself, and we should focus on making power spread out”. It feels a really important point to me. Does that make sense?
The problem with this is that power imbalances were also what made some important good things happen, so without (and if we don’t come up with a replacement mechanism) we can also get into big trouble. Robin Hanson has been talking about some aspects of this under “cultural drift”.
The most salient example (to me) of this is the spread of analytic philosophy. Without the British Empire (or the West as a whole) taking over or spreading its culture by force over large parts of the world (including by others imitating Western culture in hopes of avoiding conquest, like Japan, and China to a lesser extent), the number of people with a reasonable philosophical tradition would be even tinier than it is today.
(One could tell a plausible story about this, that good philosophy leads to good science which leads to greater military power. Or that power competition helps cull harmful memetic parasites.)
Of course this is overall terrible, and not something I endorse, but neither would I endorse an intervention to get rid of power imbalances, without figuring out how to solve this kind of problem first. (Also, to be clear, I do not think we should rest our hopes of future civilization being philosophical competent on such power dynamics. It seems like a very risky / non-robust way for that to happen, and my point is just that we could make things even worse by getting rid of it and not having a better alternative in place.)
I’m a fan of competition’s benefits as much as the next guy, but it seems to me that extreme imbalance of power actually isn’t that good for competition. Monopoly can lead to stagnation too. The optimal rate of competition and improvement probably happens under moderate imbalance.
I role to disbelieve here. He just made conquest on a wim without an ideological belief that conqiest was good?