# You are way more fallible than you think

Yes, you. (And me, of course, including everything I write here.)

Epistemic status: annoyed, a quick writeup.

TL;DR: And therefore you should be very suspicious of any personal probability estimates that are near or below your fallibility level: they are complete junk.

Most of us are very much uncalibrated. We give 80%+ chance of success/​completion to the next project even after failing a bunch of similar ones before. Even those of us who are well calibrated are still bad at the margins, where the probability is low (or, equivalently, high). Events we give the odds of 1% to can happen with the frequency of 20%. Or 0.01%. Remember, you are an embedded agent, and you only have a limited access to your own source code. Some examples:

• Pascal’s mugging:
“Give me five dollars, or I’ll use my magic powers from outside the Matrix to run a Turing machine that simulates and kills 3^^^^3 people.”
The usual resolutions are the discussions of bounded utility and such, while a fallible-self approach advocated here would imply that you as an agent are too noisy to make reliable calculations, and shouldn’t even try.

• Arguments of the sort “what if God parted the sky and appeared to you in person, would you believe then?”
Well, it helps to take a third person perspective there. If someone told you it happened to them, what would be your reaction? Probably that they are either lying or delusional, not that they literally witnessed this momentous event. This fallible-other logic should apply equally to the self. So if you witness an event like that, the first thing is to doubt your observations, and hopefully check yourself into the nearest psych ward to get evaluated for a psychotic episode.

• For a more controversial and less central example, consider strong and very strong longtermism. Despite the arguments to the contrary, it is very hard to avoid privileging a few pet scenarios because of the availability heuristic, and assign non-negligible probabilities to them. In this neck of the woods it is the AGI takeover, in the prog community it might be the climate change wiping out almost all of humanity, in some evangelical circles it can be hastening the rapture.

Now, to address the but-whatab-autists like myself, there are definitely cases where estimated tiny probabilities can be pretty accurate, like, say, playing the lottery, and it is worth calculating actions based on them. Even there you ought to be very skeptical that you have found a way to circumvent your own fallibility. There might also be hail-mary situations where the alternatives to relying on a low-probability high-payoff event are even worse, which is a premise of many a fiction story. But in general, it is always worth remembering that you are an unreliable sack of meat optimized for propagating your genes, not for logical thinking, and your odds of success in estimating small probabilities are generally lower than the estimates themselves.

• Statements like “John thinks its raining and it is not raining” and “I think it is raining and it is not raining” are not always exchangeable. Would you literally not believe your eyes if you saw something you could not explain? I also think it is easy to believe that even a third person subject had the experience but whether it is mapping anything or is just a disconnected doodle is up for grabs.

• Regarding disbelief of your own senses: As I’ve commented before: winners of mega lotteries routinely tripple check thier tickets, get 3rd party verification, and even while sitting in thier new mansion will express thier disbelief at winning.

• This is one of the main themes of Nassim Taleb’s books. You can’t really predict the future and you especially can’t predict improbable things, so minimize left-tail risk, maximize your exposure to right-tail events, and hope for the best.

• Common question: “Well, but what if God was real and actually appeared to you in flame and glory, wouldn’t it be silly to not be convinced in that case?”

My answer: “I don’t know, do you think my thought patterns are likely to be deployed in such an environment?”

• That’s another way to look at it. The usual implicit assumptions break down on the margins. Though, given the odds of this happening (once in a bush, at best, and the flame was not all that glorious), I would bet on hallucinations as a much likelier explanation. Happens to people quite often.

• How about historical precedent? Gods did arrive on Spanish ships to South America.

• Can you add some more explanation of what ones “fallibility level” might be?

• It’s a good question. If you ever do, say, project estimates at work, and look back at your track record, most of us would give 99% odds of completion to some project within a given time (well padded to make it that high), and still notice that we go over time and/​or over budget way more often than that. There are exceptions, but in general we suck at taking into account long tails.

• This fallible-other logic should apply equally to the self.

Why?

• Uh… because belief feels like truth from the inside, and so you cannot trust the inside view unless you are extremely well calibrated on tiny probabilities? So all you are left with is the outside view. If that is what you are asking.

• I think you should distrust the other person for other reasons. They may have ulterior motives, for example—as you wrote, they may be lying. There are many social reasons for them to make claims like that. Whatever lies we tell ourselves, they are usually not conscious and explicit.

Which is why I found it strange that you stated this claim without further explanation.