Though any belief so extreme wouldn’t really feel like a “belief” in the colloquial sense, I don’t internally label my belief that there is a chair under my butt as a “belief”. That label instinctually gets used for things I am much less certain about, so most normal people doing an internal search for “beliefs” will only think of things that they are not extremely certain of. Most beliefs worth having are extreme, but most beliefs internally labeled as “belief” worth having are not extreme.
“Worth having” is a separate argument about relative value of new information. It is reasonable when markets exist or we are competing in other ways where we can exploit our relative advantage. But there’s a different mistake that is possible which I want to note.
Most extreme beliefs are false; for every correct belief, there are many, many extreme beliefs that are false. Strong consensus on some belief is (evidence for the existence of) strong evidence of the truth of that belief, at least among the considered alternatives. So picking a belief on the basis of extremity (“Most sheeple think X, so consider Y”) is doing this the wrong way around, because extremity alone is negligible evidence of value. (Prosecutor’s fallacy.)
What makes the claim that extremity isn’t a useful indicator of value, less valid? That is, where should we think that extreme beliefs should even be considered?
I think the answer is when the evidence is both novel and cumulatively outweighs the prior consensus, or the belief is new / previously unconsidered. (“We went to the moon to inspect the landing site,” not “we watched the same video again and it’s clearly fake.”) So we should only consider extreme beliefs, even on the basis of our seemingly overwhelming evidence, if the proposed belief is significantly newer than the extant consensus AND we have a strong argument that the evidence is not yet widely shared / understood.
You seem to have made two logical errors here. First, “This belief is extreme” does not imply “This belief is true”, but neither does it imply “This belief is false”. You shouldn’t divide beliefs into “extreme” and “non-extreme” buckets and treat them differently.
Second, you seem to be using “extreme” to mean both “involving very high confidence” and “seen as radical”, the latter of which you might mean to be “in favour of a proposition I assign a very low prior probability”.
Restating my first objection, “This belief has prior odds of 1:1024” is exactly 10 bits of evidence against the belief. You can’t use that information to update the probability downward, because −10 bits is “extreme”, any more than you can update the probability upward because −10 bits is “extreme”. If you could do that, you would have a prior that immediately requires updating based on its own content (so it’s not your real prior), and I’m pretty sure you would either get stuck in infinite loops of lowering and raising the probability of some particular belief (based on whether it is “extreme” or not), or else be able to pump out infinite evidence for or against some belief.
Extreme, in this context, was implying far from the consensus expectation. That implies both “seen as radical” and “involving very high [consensus] confidence [against the belief].”
Contra your first paragraph, I think, I claim that this “extremeness” is valid Bayesian evidence for it being false, in the sense that you identify in your third paragraph—it has low prior odds. Given that, I agree that it would be incorrect to double-count the evidence of being extreme. But my claim was that, holding “extremeness” constant, the newness of a claim was independent reason to consider it as otherwise more worthy of examination, (rather than as more likely,) since VoI was higher / the consensus against it is less informative. And that’s why it doesn’t create a loop in the way you suggested.
So I wasn’t clear in my explanation, and thanks for trying to clarify what I meant. I hope this explains better / refined my thinking to a point where it doesn’t have the problem you identified—but if I’m still not understanding your criticism, feel free to try again.
Corollary: most beliefs worth having are extreme.
Though any belief so extreme wouldn’t really feel like a “belief” in the colloquial sense, I don’t internally label my belief that there is a chair under my butt as a “belief”. That label instinctually gets used for things I am much less certain about, so most normal people doing an internal search for “beliefs” will only think of things that they are not extremely certain of. Most beliefs worth having are extreme, but most beliefs internally labeled as “belief” worth having are not extreme.
“Worth having” is a separate argument about relative value of new information. It is reasonable when markets exist or we are competing in other ways where we can exploit our relative advantage. But there’s a different mistake that is possible which I want to note.
Most extreme beliefs are false; for every correct belief, there are many, many extreme beliefs that are false. Strong consensus on some belief is (evidence for the existence of) strong evidence of the truth of that belief, at least among the considered alternatives. So picking a belief on the basis of extremity (“Most sheeple think X, so consider Y”) is doing this the wrong way around, because extremity alone is negligible evidence of value. (Prosecutor’s fallacy.)
What makes the claim that extremity isn’t a useful indicator of value, less valid? That is, where should we think that extreme beliefs should even be considered?
I think the answer is when the evidence is both novel and cumulatively outweighs the prior consensus, or the belief is new / previously unconsidered. (“We went to the moon to inspect the landing site,” not “we watched the same video again and it’s clearly fake.”) So we should only consider extreme beliefs, even on the basis of our seemingly overwhelming evidence, if the proposed belief is significantly newer than the extant consensus AND we have a strong argument that the evidence is not yet widely shared / understood.
You seem to have made two logical errors here. First, “This belief is extreme” does not imply “This belief is true”, but neither does it imply “This belief is false”. You shouldn’t divide beliefs into “extreme” and “non-extreme” buckets and treat them differently.
Second, you seem to be using “extreme” to mean both “involving very high confidence” and “seen as radical”, the latter of which you might mean to be “in favour of a proposition I assign a very low prior probability”.
Restating my first objection, “This belief has prior odds of 1:1024” is exactly 10 bits of evidence against the belief. You can’t use that information to update the probability downward, because −10 bits is “extreme”, any more than you can update the probability upward because −10 bits is “extreme”. If you could do that, you would have a prior that immediately requires updating based on its own content (so it’s not your real prior), and I’m pretty sure you would either get stuck in infinite loops of lowering and raising the probability of some particular belief (based on whether it is “extreme” or not), or else be able to pump out infinite evidence for or against some belief.
Extreme, in this context, was implying far from the consensus expectation. That implies both “seen as radical” and “involving very high [consensus] confidence [against the belief].”
Contra your first paragraph, I think, I claim that this “extremeness” is valid Bayesian evidence for it being false, in the sense that you identify in your third paragraph—it has low prior odds. Given that, I agree that it would be incorrect to double-count the evidence of being extreme. But my claim was that, holding “extremeness” constant, the newness of a claim was independent reason to consider it as otherwise more worthy of examination, (rather than as more likely,) since VoI was higher / the consensus against it is less informative. And that’s why it doesn’t create a loop in the way you suggested.
So I wasn’t clear in my explanation, and thanks for trying to clarify what I meant. I hope this explains better / refined my thinking to a point where it doesn’t have the problem you identified—but if I’m still not understanding your criticism, feel free to try again.
FWIW, my interpretation of Eliezer’s comment was just that he meant high confidence.