There’s a big difference between summarizing data with a probability distribution, and practically flushing it away by thresholding it. The former is an associative, composable operation: You can keep doing it without really committing to any particular application of the data. The latter hands you a terminal object which is useless in the face of further information.
You don’t flush away your data, you are just running a request to it. You are not exterminating your own brain, you are only forming a categorical judgment. Like with tigers.
Sure. As I noted in the original post and also in response to your comments, sometimes it’s fine to do this. Sometimes it isn’t.
You are throwing away information when you threshold, period, end of that discussion. If that information was important, you made a mistake. It is easy to identify these situations because they have aberrant expected values in low-probability outcomes. And it is easy to avoid them, unlike with tigers.
You are throwing away information when you threshold, period, end of that discussion.
If the ‘information’ was noise, then it’s not information, it’s just data. Whether it counts as information depends upon relevance, which is partially what the threshold is for.
Neither, you just take a hit according to your scoring rule; but if you’re properly calibrated, it’ll be compensated for on average by 99 times that your 1-in-a-hundred chances don’t happen.
If I repeatedly claim I have a 1-in-2 chance of rolling snake eyes, you’d probably see me take repeated blows by a logarithmic scoring rule, which would suffice for bystanders to put less trust in my probability estimates. Eventually I should admit I’m poorly calibrated.
Of course if you’re talking about a probability 1 minus epsilon threshold, a miss on that claim is a huge penalty by a log scoring rule.
There’s a big difference between summarizing data with a probability distribution, and practically flushing it away by thresholding it. The former is an associative, composable operation: You can keep doing it without really committing to any particular application of the data. The latter hands you a terminal object which is useless in the face of further information.
You don’t flush away your data, you are just running a request to it. You are not exterminating your own brain, you are only forming a categorical judgment. Like with tigers.
Sure. As I noted in the original post and also in response to your comments, sometimes it’s fine to do this. Sometimes it isn’t.
You are throwing away information when you threshold, period, end of that discussion. If that information was important, you made a mistake. It is easy to identify these situations because they have aberrant expected values in low-probability outcomes. And it is easy to avoid them, unlike with tigers.
If the ‘information’ was noise, then it’s not information, it’s just data. Whether it counts as information depends upon relevance, which is partially what the threshold is for.
Link for more info
Of course, this is a jargon use in information theory. But that seems like the relevant domain.
It’s not useless, not by any means.
A threshold can be contradicted by a contrary-to-expectation observation. A statement of probability cannot, as long as it’s not an absolute.
If you believe that there’s only one-in-a-hundred chance of something happening, and it happens, were you right or wrong?
Neither, you just take a hit according to your scoring rule; but if you’re properly calibrated, it’ll be compensated for on average by 99 times that your 1-in-a-hundred chances don’t happen.
If I repeatedly claim I have a 1-in-2 chance of rolling snake eyes, you’d probably see me take repeated blows by a logarithmic scoring rule, which would suffice for bystanders to put less trust in my probability estimates. Eventually I should admit I’m poorly calibrated.
Of course if you’re talking about a probability 1 minus epsilon threshold, a miss on that claim is a huge penalty by a log scoring rule.