Oh what a tangled web we weave, when first we practice to deceive.
The more the algorithm probes the deception, the more effort the deceiver has to put into maintaining the illusion. Reality is completely self-consistent, so the accumulation of enough evidence into a sufficiently coherent world model will always resolve these types of deceptions if no further effort is applied to maintain them.
When I think of deception, though, I typically imagine a deceiver trying to create a strategic mismatch between their beliefs and another agent’s beliefs. This information asymmetry is typically over something that the deceiver does not expect the agent to be able to investigate easily. It then allows them to get away with something that is in the deceiver’s interest but against (what they think is probably) the other agent’s interest.
It’s usually something like a claim about the health benefits of snake oil (where the agent lacks the time or the resources to perform or look up a randomized controlled trial), rather than a claim about the existence of a physical object that’s visible and within reach.
Maybe magic tricks go that route, though. Although those sorts of deceptions seem to be more about creating a superstimulus for the audience’s curiosity drive, since the audience knows that their senses are being deceived and yet go out of their way to experience that deception.
The teapot example is atypical of deception in humans, and was chosen to be simple and clear-cut. I think the web-of-lies effect is hampered in humans by a couple of things, both of which result from us only being approximations of Bayesian reasoners. One is the limits to our computation, we can’t go and check a new update that “snake oil works” against all possible connections. Another part (which is also linked to computation limits) is that I suspect a small enough discrepancy gets rounded down to zero.
So if I’m convinced that “snake oil is effective against depression”. I don’t necessarily check it against literally all the beliefs I have about depression, which limits the spread of the web. Or if it only very slightly contradicts my existing view of the mechanism of depression, that won’t be enough for me to update the existing view at all, and the difference is swept under the rug. So the web peters out.
Of course the main reason snake oil salesmen work is because they play into people’s existing biases.
But perhaps more importantly:
This information asymmetry is typically over something that the deceiver does not expect the agent to be able to investigate easily.
This to me seems like regions where the function world→model(world) just isn’t defined yet, or is very fuzzy. This means rather than a web of lies we have some lies isolated from the rest of the model by a region of confusion. This means there is no discontinuity in the function, which might be an issue.
This reminds me of the proverb:
The more the algorithm probes the deception, the more effort the deceiver has to put into maintaining the illusion. Reality is completely self-consistent, so the accumulation of enough evidence into a sufficiently coherent world model will always resolve these types of deceptions if no further effort is applied to maintain them.
When I think of deception, though, I typically imagine a deceiver trying to create a strategic mismatch between their beliefs and another agent’s beliefs. This information asymmetry is typically over something that the deceiver does not expect the agent to be able to investigate easily. It then allows them to get away with something that is in the deceiver’s interest but against (what they think is probably) the other agent’s interest.
It’s usually something like a claim about the health benefits of snake oil (where the agent lacks the time or the resources to perform or look up a randomized controlled trial), rather than a claim about the existence of a physical object that’s visible and within reach.
Maybe magic tricks go that route, though. Although those sorts of deceptions seem to be more about creating a superstimulus for the audience’s curiosity drive, since the audience knows that their senses are being deceived and yet go out of their way to experience that deception.
You make some really excellent points here.
The teapot example is atypical of deception in humans, and was chosen to be simple and clear-cut. I think the web-of-lies effect is hampered in humans by a couple of things, both of which result from us only being approximations of Bayesian reasoners. One is the limits to our computation, we can’t go and check a new update that “snake oil works” against all possible connections. Another part (which is also linked to computation limits) is that I suspect a small enough discrepancy gets rounded down to zero.
So if I’m convinced that “snake oil is effective against depression”. I don’t necessarily check it against literally all the beliefs I have about depression, which limits the spread of the web. Or if it only very slightly contradicts my existing view of the mechanism of depression, that won’t be enough for me to update the existing view at all, and the difference is swept under the rug. So the web peters out.
Of course the main reason snake oil salesmen work is because they play into people’s existing biases.
But perhaps more importantly:
This to me seems like regions where the function world→model(world) just isn’t defined yet, or is very fuzzy. This means rather than a web of lies we have some lies isolated from the rest of the model by a region of confusion. This means there is no discontinuity in the function, which might be an issue.