Note that in the original footnote in my post, “on the same side” is a hyperlink going to a comment by Val
Thanks for pointing this out. (I read Val’s comment while writing my post, but unfortunately neglected to add the hyperlink when pasting the text of the footnote into my draft.) I have now edited the link into my post.
the goal isn’t to trick people into thinking your disagreements are small, it’s to make typical disagreements feel less like battles between warring armies
I think the fact that disagreements often feel like battles between warring armies is because a lot of apparent “disagreements” are usefully modeled as disguised conflicts. That is, my theory about why predictable disagreements are so ubiquitous in human life (despite the fact that Bayesian reasoners can’t forsee to disagree) is mostly conflict-theoretic rather than mistake-theoretic.
A simple example: I stole a loaf of bread. A policeman thinks I stole the bread. I claim that I didn’t steal the bread. Superficially, this looks like a “disagreement” to an outside observer noticing the two of us reporting different beliefs, but what’s actually going on is that I’m lying. Importantly, if I care more about not going to jail than I do about being honest, lying is rational. Agents have an incentive to build maps that reflect the territory because those are the maps that are most useful for computing effective plans … but they also sometimes have an incentive to sabotage the maps of other agents with different utility functions.
Most interesting real-world disagreements aren’t so simple as the “one party is lying” case. But I think the moral should generalize: predictable disagreements are mostly due to at least some parts of some parties’ brains trying to optimize for conflicting goals, rather than just being “innocently” mistaken.
I’m incredibly worried that approaches to “cooperative” or “collaborative truth-seeking” that try to cultivate the spirit that everyone is on the same side and we all just want to get to the truth, quickly collapse in practice to, “I’ll accept your self-aggrandizing lies, if you accept my self-aggrandizing lies”—not because anyone thinks of themselves as telling self-aggrandizing lies, but because that’s what the elephant in the brain does by default. I’m more optimistic about approaches that are open to the possibility that conflicts exist, in the hopes that exposing hidden conflicts (rather than pretending they’re “disagreements”) makes it easier to find Pareto improvements.
I’m incredibly worried that approaches to “cooperative” or “collaborative truth-seeking” that try to cultivate the spirit that everyone is on the same side and we all just want to get to the truth, quickly collapse in practice to, “I’ll accept your self-aggrandizing lies, if you accept my self-aggrandizing lies”—not because anyone thinks of themselves as telling self-aggrandizing lies, but because that’s what the elephant in the brain does by default.
Very strongly seconding this. (I have noticed this pattern on Less Wrong in the past, in fact, and more than once. It is no idle worry, but a very real thing that already happens.)
Thanks for pointing this out. (I read Val’s comment while writing my post, but unfortunately neglected to add the hyperlink when pasting the text of the footnote into my draft.) I have now edited the link into my post.
I think the fact that disagreements often feel like battles between warring armies is because a lot of apparent “disagreements” are usefully modeled as disguised conflicts. That is, my theory about why predictable disagreements are so ubiquitous in human life (despite the fact that Bayesian reasoners can’t forsee to disagree) is mostly conflict-theoretic rather than mistake-theoretic.
A simple example: I stole a loaf of bread. A policeman thinks I stole the bread. I claim that I didn’t steal the bread. Superficially, this looks like a “disagreement” to an outside observer noticing the two of us reporting different beliefs, but what’s actually going on is that I’m lying. Importantly, if I care more about not going to jail than I do about being honest, lying is rational. Agents have an incentive to build maps that reflect the territory because those are the maps that are most useful for computing effective plans … but they also sometimes have an incentive to sabotage the maps of other agents with different utility functions.
Most interesting real-world disagreements aren’t so simple as the “one party is lying” case. But I think the moral should generalize: predictable disagreements are mostly due to at least some parts of some parties’ brains trying to optimize for conflicting goals, rather than just being “innocently” mistaken.
I’m incredibly worried that approaches to “cooperative” or “collaborative truth-seeking” that try to cultivate the spirit that everyone is on the same side and we all just want to get to the truth, quickly collapse in practice to, “I’ll accept your self-aggrandizing lies, if you accept my self-aggrandizing lies”—not because anyone thinks of themselves as telling self-aggrandizing lies, but because that’s what the elephant in the brain does by default. I’m more optimistic about approaches that are open to the possibility that conflicts exist, in the hopes that exposing hidden conflicts (rather than pretending they’re “disagreements”) makes it easier to find Pareto improvements.
Very strongly seconding this. (I have noticed this pattern on Less Wrong in the past, in fact, and more than once. It is no idle worry, but a very real thing that already happens.)