Why does prediction-book 1) allow you to make 100% credence predictions and 2) bucket 99% credence in the 90% bucket instead of the 100% bucket?
Does anyone know?
It means I either need to have an unsightly graph where my measured accuracy falls to 0 in the 100% bucket, or take the unseemly approach of putting 100% (rounding up, of course, not literally 100%) on some extremely likely prediction.
The bucketing also means that if I make many 99% predictions, but few 90% predictions (for instance), I’ll appear uncalibrated, even if I have perfect calibration, (since items in that bucket would be accurate more than 90% of the time). Not realizing this, I might think that I need to adjust more.
Why does prediction-book 1) allow you to make 100% credence predictions and 2) bucket 99% credence in the 90% bucket instead of the 100% bucket?
Does anyone know?
It means I either need to have an unsightly graph where my measured accuracy falls to 0 in the 100% bucket, or take the unseemly approach of putting 100% (rounding up, of course, not literally 100%) on some extremely likely prediction.
The bucketing also means that if I make many 99% predictions, but few 90% predictions (for instance), I’ll appear uncalibrated, even if I have perfect calibration, (since items in that bucket would be accurate more than 90% of the time). Not realizing this, I might think that I need to adjust more.