Yes, you. (And me, of course, including everything I write here.)
Epistemic status: annoyed, a quick writeup.
TL;DR: And therefore you should be very suspicious of any personal probability estimates that are near or below your fallibility level: they are complete junk.
Most of us are very much uncalibrated. We give 80%+ chance of success/completion to the next project even after failing a bunch of similar ones before. Even those of us who are well calibrated are still bad at the margins, where the probability is low (or, equivalently, high). Events we give the odds of 1% to can happen with the frequency of 20%. Or 0.01%. Remember, you are an embedded agent, and you only have a limited access to your own source code. Some examples:
Pascal’s mugging: “Give me five dollars, or I’ll use my magic powers from outside the Matrix to run a Turing machine that simulates and kills 3^^^^3 people.” The usual resolutions are the discussions of bounded utility and such, while a fallible-self approach advocated here would imply that you as an agent are too noisy to make reliable calculations, and shouldn’t even try.
Arguments of the sort “what if God parted the sky and appeared to you in person, would you believe then?” Well, it helps to take a third person perspective there. If someone told you it happened to them, what would be your reaction? Probably that they are either lying or delusional, not that they literally witnessed this momentous event. This fallible-other logic should apply equally to the self. So if you witness an event like that, the first thing is to doubt your observations, and hopefully check yourself into the nearest psych ward to get evaluated for a psychotic episode.
For a more controversial and less central example, consider strong and very strong longtermism. Despite the arguments to the contrary, it is very hard to avoid privileging a few pet scenarios because of the availability heuristic, and assign non-negligible probabilities to them. In this neck of the woods it is the AGI takeover, in the prog community it might be the climate change wiping out almost all of humanity, in some evangelical circles it can be hastening the rapture.
Now, to address the but-whatab-autists like myself, there are definitely cases where estimated tiny probabilities can be pretty accurate, like, say, playing the lottery, and it is worth calculating actions based on them. Even there you ought to be very skeptical that you have found a way to circumvent your own fallibility. There might also be hail-mary situations where the alternatives to relying on a low-probability high-payoff event are even worse, which is a premise of many a fiction story. But in general, it is always worth remembering that you are an unreliable sack of meat optimized for propagating your genes, not for logical thinking, and your odds of success in estimating small probabilities are generally lower than the estimates themselves.
You are way more fallible than you think
Yes, you. (And me, of course, including everything I write here.)
Epistemic status: annoyed, a quick writeup.
TL;DR: And therefore you should be very suspicious of any personal probability estimates that are near or below your fallibility level: they are complete junk.
Most of us are very much uncalibrated. We give 80%+ chance of success/completion to the next project even after failing a bunch of similar ones before. Even those of us who are well calibrated are still bad at the margins, where the probability is low (or, equivalently, high). Events we give the odds of 1% to can happen with the frequency of 20%. Or 0.01%. Remember, you are an embedded agent, and you only have a limited access to your own source code. Some examples:
Pascal’s mugging:
“Give me five dollars, or I’ll use my magic powers from outside the Matrix to run a Turing machine that simulates and kills 3^^^^3 people.”
The usual resolutions are the discussions of bounded utility and such, while a fallible-self approach advocated here would imply that you as an agent are too noisy to make reliable calculations, and shouldn’t even try.
Arguments of the sort “what if God parted the sky and appeared to you in person, would you believe then?”
Well, it helps to take a third person perspective there. If someone told you it happened to them, what would be your reaction? Probably that they are either lying or delusional, not that they literally witnessed this momentous event. This fallible-other logic should apply equally to the self. So if you witness an event like that, the first thing is to doubt your observations, and hopefully check yourself into the nearest psych ward to get evaluated for a psychotic episode.
For a more controversial and less central example, consider strong and very strong longtermism. Despite the arguments to the contrary, it is very hard to avoid privileging a few pet scenarios because of the availability heuristic, and assign non-negligible probabilities to them. In this neck of the woods it is the AGI takeover, in the prog community it might be the climate change wiping out almost all of humanity, in some evangelical circles it can be hastening the rapture.
Now, to address the but-whatab-autists like myself, there are definitely cases where estimated tiny probabilities can be pretty accurate, like, say, playing the lottery, and it is worth calculating actions based on them. Even there you ought to be very skeptical that you have found a way to circumvent your own fallibility. There might also be hail-mary situations where the alternatives to relying on a low-probability high-payoff event are even worse, which is a premise of many a fiction story. But in general, it is always worth remembering that you are an unreliable sack of meat optimized for propagating your genes, not for logical thinking, and your odds of success in estimating small probabilities are generally lower than the estimates themselves.