You seem to have made two logical errors here. First, “This belief is extreme” does not imply “This belief is true”, but neither does it imply “This belief is false”. You shouldn’t divide beliefs into “extreme” and “non-extreme” buckets and treat them differently.
Second, you seem to be using “extreme” to mean both “involving very high confidence” and “seen as radical”, the latter of which you might mean to be “in favour of a proposition I assign a very low prior probability”.
Restating my first objection, “This belief has prior odds of 1:1024” is exactly 10 bits of evidence against the belief. You can’t use that information to update the probability downward, because −10 bits is “extreme”, any more than you can update the probability upward because −10 bits is “extreme”. If you could do that, you would have a prior that immediately requires updating based on its own content (so it’s not your real prior), and I’m pretty sure you would either get stuck in infinite loops of lowering and raising the probability of some particular belief (based on whether it is “extreme” or not), or else be able to pump out infinite evidence for or against some belief.
Extreme, in this context, was implying far from the consensus expectation. That implies both “seen as radical” and “involving very high [consensus] confidence [against the belief].”
Contra your first paragraph, I think, I claim that this “extremeness” is valid Bayesian evidence for it being false, in the sense that you identify in your third paragraph—it has low prior odds. Given that, I agree that it would be incorrect to double-count the evidence of being extreme. But my claim was that, holding “extremeness” constant, the newness of a claim was independent reason to consider it as otherwise more worthy of examination, (rather than as more likely,) since VoI was higher / the consensus against it is less informative. And that’s why it doesn’t create a loop in the way you suggested.
So I wasn’t clear in my explanation, and thanks for trying to clarify what I meant. I hope this explains better / refined my thinking to a point where it doesn’t have the problem you identified—but if I’m still not understanding your criticism, feel free to try again.
You seem to have made two logical errors here. First, “This belief is extreme” does not imply “This belief is true”, but neither does it imply “This belief is false”. You shouldn’t divide beliefs into “extreme” and “non-extreme” buckets and treat them differently.
Second, you seem to be using “extreme” to mean both “involving very high confidence” and “seen as radical”, the latter of which you might mean to be “in favour of a proposition I assign a very low prior probability”.
Restating my first objection, “This belief has prior odds of 1:1024” is exactly 10 bits of evidence against the belief. You can’t use that information to update the probability downward, because −10 bits is “extreme”, any more than you can update the probability upward because −10 bits is “extreme”. If you could do that, you would have a prior that immediately requires updating based on its own content (so it’s not your real prior), and I’m pretty sure you would either get stuck in infinite loops of lowering and raising the probability of some particular belief (based on whether it is “extreme” or not), or else be able to pump out infinite evidence for or against some belief.
Extreme, in this context, was implying far from the consensus expectation. That implies both “seen as radical” and “involving very high [consensus] confidence [against the belief].”
Contra your first paragraph, I think, I claim that this “extremeness” is valid Bayesian evidence for it being false, in the sense that you identify in your third paragraph—it has low prior odds. Given that, I agree that it would be incorrect to double-count the evidence of being extreme. But my claim was that, holding “extremeness” constant, the newness of a claim was independent reason to consider it as otherwise more worthy of examination, (rather than as more likely,) since VoI was higher / the consensus against it is less informative. And that’s why it doesn’t create a loop in the way you suggested.
So I wasn’t clear in my explanation, and thanks for trying to clarify what I meant. I hope this explains better / refined my thinking to a point where it doesn’t have the problem you identified—but if I’m still not understanding your criticism, feel free to try again.
FWIW, my interpretation of Eliezer’s comment was just that he meant high confidence.