The teapot example is atypical of deception in humans, and was chosen to be simple and clear-cut. I think the web-of-lies effect is hampered in humans by a couple of things, both of which result from us only being approximations of Bayesian reasoners. One is the limits to our computation, we can’t go and check a new update that “snake oil works” against all possible connections. Another part (which is also linked to computation limits) is that I suspect a small enough discrepancy gets rounded down to zero.
So if I’m convinced that “snake oil is effective against depression”. I don’t necessarily check it against literally all the beliefs I have about depression, which limits the spread of the web. Or if it only very slightly contradicts my existing view of the mechanism of depression, that won’t be enough for me to update the existing view at all, and the difference is swept under the rug. So the web peters out.
Of course the main reason snake oil salesmen work is because they play into people’s existing biases.
But perhaps more importantly:
This information asymmetry is typically over something that the deceiver does not expect the agent to be able to investigate easily.
This to me seems like regions where the function world→model(world) just isn’t defined yet, or is very fuzzy. This means rather than a web of lies we have some lies isolated from the rest of the model by a region of confusion. This means there is no discontinuity in the function, which might be an issue.
You make some really excellent points here.
The teapot example is atypical of deception in humans, and was chosen to be simple and clear-cut. I think the web-of-lies effect is hampered in humans by a couple of things, both of which result from us only being approximations of Bayesian reasoners. One is the limits to our computation, we can’t go and check a new update that “snake oil works” against all possible connections. Another part (which is also linked to computation limits) is that I suspect a small enough discrepancy gets rounded down to zero.
So if I’m convinced that “snake oil is effective against depression”. I don’t necessarily check it against literally all the beliefs I have about depression, which limits the spread of the web. Or if it only very slightly contradicts my existing view of the mechanism of depression, that won’t be enough for me to update the existing view at all, and the difference is swept under the rug. So the web peters out.
Of course the main reason snake oil salesmen work is because they play into people’s existing biases.
But perhaps more importantly:
This to me seems like regions where the function world→model(world) just isn’t defined yet, or is very fuzzy. This means rather than a web of lies we have some lies isolated from the rest of the model by a region of confusion. This means there is no discontinuity in the function, which might be an issue.