So I’ve thought about this a bit more. It doesn’t matter how someone states their probabilities. However, in order to use your evaluation technique we just need to transform the probabilities so that all of them are above the baseline.
In any case, it’s good to see this post. I’ve always worried for a long time that being calibrated on 50% estimates mightn’t be very meaningful as you might be massively overconfident on some guesses and massively underconfident on others.
So I’ve thought about this a bit more. It doesn’t matter how someone states their probabilities. However, in order to use your evaluation technique we just need to transform the probabilities so that all of them are above the baseline.
In any case, it’s good to see this post. I’ve always worried for a long time that being calibrated on 50% estimates mightn’t be very meaningful as you might be massively overconfident on some guesses and massively underconfident on others.