Self-interested agents will rationally resist coercion even at some cost to themselves, because this means other agents are incentivized not to threaten them. So evolution has implemented a hacky version of this logic in humans, and people who have never thought about game theory often find themselves hating their enemies and plotting to avenge their wrongs, even if that means costly fighting rather than an unequal peace
I think that Alex M. has done some pretty good research on this; e.g. even just over the last 1000 years in Europe, people who were good at ideological conformity in the religious environment (e.g. quick understanding of Pascal’s-wager-like arguments without the aid of literacy, but either don’t develop traits that allow Pascal’s-wager-like arguments allow clever speakers to take their resources or dissuading them from maximizing offspring, or developing counter-traits e.g. causing them to randomly just maximize offspring anyway). But it’s pretty clear that the last 10,000 years of civilization probably had something to do with people’s tendency to choose social reality over objective reality, since locking up and refusing to cooperate with a line of logic is a great way to survive in a genepool where some people are smarter than others and think up galaxy-brained coercion strategies.
some sort of actor which threatens humanity with negative consequences (e.g. torturing simulated copies of us) for refusing to give it resources, but must spend its own valuable resources to do so.
This is one of the big reasons why it’s so ridiculous what the DoD was doing with the flying saucer videos, leaving it open-ended whether they think they were human, they clearly have no idea what they’re doing (e.g. maybe they brought in a rando from SETI or something). The impression I get is that they and others in the Natsec community are generally competent enough to understand that if there were aliens, there would be quadrillions of ships, and maybe that they would probably be as smart to humans as humans are to ants. But they probably aren’t competent enough to understand that the question of whether aliens would torture humans in order to maximize compliance, is a math question; let alone matters of realityfluid and simulated copies or other things that would actually happen if you actually encountered an opposing force that was as smart to you as you were to ants (Yudkowsky recently put the odds of friendliness/benign at 5%). The cluelessness on the DoD’s part was just embarrassing; it’s not rocket science, and they don’t just have tons of rocket scientists, their organization is the direct deliberate cause of why so many rocket scientists were pumped out in the first place.
I think that Alex M. has done some pretty good research on this; e.g. even just over the last 1000 years in Europe, people who were good at ideological conformity in the religious environment (e.g. quick understanding of Pascal’s-wager-like arguments without the aid of literacy, but either don’t develop traits that allow Pascal’s-wager-like arguments allow clever speakers to take their resources or dissuading them from maximizing offspring, or developing counter-traits e.g. causing them to randomly just maximize offspring anyway). But it’s pretty clear that the last 10,000 years of civilization probably had something to do with people’s tendency to choose social reality over objective reality, since locking up and refusing to cooperate with a line of logic is a great way to survive in a genepool where some people are smarter than others and think up galaxy-brained coercion strategies.
This is one of the big reasons why it’s so ridiculous what the DoD was doing with the flying saucer videos, leaving it open-ended whether they think they were human, they clearly have no idea what they’re doing (e.g. maybe they brought in a rando from SETI or something). The impression I get is that they and others in the Natsec community are generally competent enough to understand that if there were aliens, there would be quadrillions of ships, and maybe that they would probably be as smart to humans as humans are to ants. But they probably aren’t competent enough to understand that the question of whether aliens would torture humans in order to maximize compliance, is a math question; let alone matters of realityfluid and simulated copies or other things that would actually happen if you actually encountered an opposing force that was as smart to you as you were to ants (Yudkowsky recently put the odds of friendliness/benign at 5%). The cluelessness on the DoD’s part was just embarrassing; it’s not rocket science, and they don’t just have tons of rocket scientists, their organization is the direct deliberate cause of why so many rocket scientists were pumped out in the first place.