Neither, you just take a hit according to your scoring rule; but if you’re properly calibrated, it’ll be compensated for on average by 99 times that your 1-in-a-hundred chances don’t happen.
If I repeatedly claim I have a 1-in-2 chance of rolling snake eyes, you’d probably see me take repeated blows by a logarithmic scoring rule, which would suffice for bystanders to put less trust in my probability estimates. Eventually I should admit I’m poorly calibrated.
Of course if you’re talking about a probability 1 minus epsilon threshold, a miss on that claim is a huge penalty by a log scoring rule.
It’s not useless, not by any means.
A threshold can be contradicted by a contrary-to-expectation observation. A statement of probability cannot, as long as it’s not an absolute.
If you believe that there’s only one-in-a-hundred chance of something happening, and it happens, were you right or wrong?
Neither, you just take a hit according to your scoring rule; but if you’re properly calibrated, it’ll be compensated for on average by 99 times that your 1-in-a-hundred chances don’t happen.
If I repeatedly claim I have a 1-in-2 chance of rolling snake eyes, you’d probably see me take repeated blows by a logarithmic scoring rule, which would suffice for bystanders to put less trust in my probability estimates. Eventually I should admit I’m poorly calibrated.
Of course if you’re talking about a probability 1 minus epsilon threshold, a miss on that claim is a huge penalty by a log scoring rule.