Epistemic status: Boggling. This is early, messy work.
Thanks to Edo Arad for comments and suggestions.
Part 1: A Brief Tale
You’re exploring a vast land filled with forests and brush. You thrash your sword to carve out a path and make sense of things.
There are monsters. Lovecraftian winged monstrosities. Their attacks damage your sanity. When injured, you don’t notice, but your mind will begin to wander. You begin seeing things that aren’t real.
You have a shield that protects you from the beasts. You have armor as well. They attack; you deflect. You hear a thud, a squeal, then light flapping sounds when they retreat. Occasionally you can even catch them with your sword but they move quickly.
You return to the village. Your companions have returned too. Some of them aren’t right. They’re obsessed with bizarre imaginings of angels and gods. Some speak of finding ancient labyrinths in places you know are full of swamps.
Your companions size each other up. Factions emerge. One cluster suggests that the monsters bring forward wisdom. The monsters should be brought directly into the village and released.
The pro-monster faction is visibly scratched up; they must have been wounded. They deny taking damage, that the marks on their bodies were extraneous. Some hallucinate wearing a great deal of armor, even though they are visibly unprotected. They do however possess large weapons (likely a trade-off from having light armor), so others are nervous about disagreeing.
The bulk of the crowd turns against this faction. Some attack back; a few people are harmed, but in the end, the faction loses. Some are thrown into jails, others delegated to the very safest of tasks. But this cycle will repeat. You might not be so lucky next time.
The village learns to cope. There are vast regions of bush and thorn but no monsters. Soldiers with little armor are sent there. Some dual wield axes or use two-handed broadswords.
Other areas have many monsters but plain land. This is where you send your fully plated knights. They’ll move slowly but have the best chances of keeping their sanity.
Part 2: An Explanation
The topic is intelligence, epistemics, and sanity.
Weapons are raw intellect, cleverness, narrow technical abilities. These are powerful but dangerous.
Shields and armor represent epistemics, wisdom, humility, calibration. Not believing false things. Not going functionally insane. Winning the intellectual war, though perhaps losing the battle.
The monsters are epistemic hazards that lead good people to believe bad things. See: Politics is the Mind-Killer. Pure math is (relatively) safe.[1] Few people learn new math proofs and proceed to imagine ridiculous things about epistemology. But politics, social sciences, news, and religion, are quite treacherous. They’re full of gradients that cause people to believe false things. These fields have incentives that encourage dramatic overconfidence and motivated reasoning.
Mein Kampf is a token example. For people with solid epistemics and who are otherwise well-read, reading Mein Kampf can be enlightening. It’s necessary reading to deeply understand what exactly happened in Hitler’s rise to power. But there are clearly other readers who would come away with exactly the wrong lessons, and wind up becoming sympathetic to Adolf Hitler. Mein Kampf represents dangerous territory to these people.
Televised news and social media news are both highly hazardous. People who learn from these sources and often become extremists of one form or another. We haven’t found easy ways to give them the right defenses; to allow them to come from these resources as constantly more correct about the world, instead of less.
Individuals with powerful intellectual weapons but weak epistemics can be extremely dangerous. Consider all of the highly well-coordinated destruction caused by certain religious groups, or, on an individual level, vocal advocates of these groups.
Individuals with great epistemics but weak intellect are safe, but often not very useful. Think of soft-spoken people who are poor at arguing. It takes great effort to discern these folks and listen to them accordingly.
Increases in intellect can be dangerous. Increases in epistemics are occasionally so. If your enemies have improved epistemics, but still don’t see your point of view, that could well be a bad thing. The scariest threat is a force with brutal intelligence and epistemics that are excellent except for one crucial detail.
Sanity in the tale is just that; sanity in real life. Perhaps we might not call crazed conspiracy theorists *insane*, but I hope we can admit they often have low *sanity*. At some point, fundamental incorrect beliefs infect a great deal of one’s worldview and lifestyle. When you take epistemic damage, from epistemic hazards, you lose sanity. Losing sanity lessens your epistemics, sometimes leading to self-reinforcing cycles.
Part 3: A Dungeons and Dragons Variant
Amelia has high intelligence, medium sanity, and medium epistemics.
She reads an epistemically dangerous article about Holocaust denialism. Her epistemic level is taken into account, and she must roll a 6-sided die.
5-6: She understands false news better. +1 to epistemics.
4: She glances over it. No effect.
2-3: She’s a tiny bit convinced. −1 to sanity.
1: She’s deeply convinced. −3 to sanity, −1 to epistemics.
Amelia rolls a 2.
Much later, Amelia is interested in writing a public blog post of her choosing. Her new intelligence, sanity, and epistemics are taken into account. A die is rolled.
Enlightened Utility Points are used as the victory points in this game. It’s a concept similar to CEV. It’s meant to represent what Amelia would value if she were completely sane and intelligent.
4-6: She writes it about a generally reasonable topic. +1 Enlightened Utility Point
2-3: She writes a post presenting some evidence for Holocaust denialism. −2 Enlightened Utility Points
1: She comes up with novel and effective arguments that will greatly help the Holocaust denialism side. −20 Enlightened Utility Points.
Amelia rolls a 4.
----
[1] To be specific, pure math is mostly safe from epistemic hazards. However, I would classify it more as an intelligence field than an epistemics field. Organizations with poor sanity might use mathematical knowledge in harmful ways.
Intelligence, epistemics, and sanity, in three short parts
Epistemic status: Boggling. This is early, messy work.
Thanks to Edo Arad for comments and suggestions.
Part 1: A Brief Tale
You’re exploring a vast land filled with forests and brush. You thrash your sword to carve out a path and make sense of things.
There are monsters. Lovecraftian winged monstrosities. Their attacks damage your sanity. When injured, you don’t notice, but your mind will begin to wander. You begin seeing things that aren’t real.
You have a shield that protects you from the beasts. You have armor as well. They attack; you deflect. You hear a thud, a squeal, then light flapping sounds when they retreat. Occasionally you can even catch them with your sword but they move quickly.
You return to the village. Your companions have returned too. Some of them aren’t right. They’re obsessed with bizarre imaginings of angels and gods. Some speak of finding ancient labyrinths in places you know are full of swamps.
Your companions size each other up. Factions emerge. One cluster suggests that the monsters bring forward wisdom. The monsters should be brought directly into the village and released.
The pro-monster faction is visibly scratched up; they must have been wounded. They deny taking damage, that the marks on their bodies were extraneous. Some hallucinate wearing a great deal of armor, even though they are visibly unprotected. They do however possess large weapons (likely a trade-off from having light armor), so others are nervous about disagreeing.
The bulk of the crowd turns against this faction. Some attack back; a few people are harmed, but in the end, the faction loses. Some are thrown into jails, others delegated to the very safest of tasks. But this cycle will repeat. You might not be so lucky next time.
The village learns to cope. There are vast regions of bush and thorn but no monsters. Soldiers with little armor are sent there. Some dual wield axes or use two-handed broadswords.
Other areas have many monsters but plain land. This is where you send your fully plated knights. They’ll move slowly but have the best chances of keeping their sanity.
Part 2: An Explanation
The topic is intelligence, epistemics, and sanity.
Weapons are raw intellect, cleverness, narrow technical abilities. These are powerful but dangerous.
Shields and armor represent epistemics, wisdom, humility, calibration. Not believing false things. Not going functionally insane. Winning the intellectual war, though perhaps losing the battle.
The monsters are epistemic hazards that lead good people to believe bad things. See: Politics is the Mind-Killer. Pure math is (relatively) safe.[1] Few people learn new math proofs and proceed to imagine ridiculous things about epistemology. But politics, social sciences, news, and religion, are quite treacherous. They’re full of gradients that cause people to believe false things. These fields have incentives that encourage dramatic overconfidence and motivated reasoning.
Mein Kampf is a token example. For people with solid epistemics and who are otherwise well-read, reading Mein Kampf can be enlightening. It’s necessary reading to deeply understand what exactly happened in Hitler’s rise to power. But there are clearly other readers who would come away with exactly the wrong lessons, and wind up becoming sympathetic to Adolf Hitler. Mein Kampf represents dangerous territory to these people.
Televised news and social media news are both highly hazardous. People who learn from these sources and often become extremists of one form or another. We haven’t found easy ways to give them the right defenses; to allow them to come from these resources as constantly more correct about the world, instead of less.
Individuals with powerful intellectual weapons but weak epistemics can be extremely dangerous. Consider all of the highly well-coordinated destruction caused by certain religious groups, or, on an individual level, vocal advocates of these groups.
Individuals with great epistemics but weak intellect are safe, but often not very useful. Think of soft-spoken people who are poor at arguing. It takes great effort to discern these folks and listen to them accordingly.
Increases in intellect can be dangerous. Increases in epistemics are occasionally so. If your enemies have improved epistemics, but still don’t see your point of view, that could well be a bad thing. The scariest threat is a force with brutal intelligence and epistemics that are excellent except for one crucial detail.
Sanity in the tale is just that; sanity in real life. Perhaps we might not call crazed conspiracy theorists *insane*, but I hope we can admit they often have low *sanity*. At some point, fundamental incorrect beliefs infect a great deal of one’s worldview and lifestyle. When you take epistemic damage, from epistemic hazards, you lose sanity. Losing sanity lessens your epistemics, sometimes leading to self-reinforcing cycles.
Part 3: A Dungeons and Dragons Variant
Amelia has high intelligence, medium sanity, and medium epistemics.
She reads an epistemically dangerous article about Holocaust denialism. Her epistemic level is taken into account, and she must roll a 6-sided die.
Amelia rolls a 2.
Much later, Amelia is interested in writing a public blog post of her choosing. Her new intelligence, sanity, and epistemics are taken into account. A die is rolled.
Enlightened Utility Points are used as the victory points in this game. It’s a concept similar to CEV. It’s meant to represent what Amelia would value if she were completely sane and intelligent.
Amelia rolls a 4.
----
[1] To be specific, pure math is mostly safe from epistemic hazards. However, I would classify it more as an intelligence field than an epistemics field. Organizations with poor sanity might use mathematical knowledge in harmful ways.