The vast majority of yes/no questions you’re likely to face won’t support 5% intervals. You’re just not going to get enough data to have any idea whether the “true” calibration is what actually happens for that small selection of questions.
That said, I agree there’s an analytic flaw if you can change true to false on no additional data (kind of: you noticed salience of something you’d previously ignored, which may count as evidence depending on how you arrived at your prior) and only reduce confidence a tiny amount.
One suggestion that may help: don’t separate your answer from your confidence confidence, just calculate a probability. Not “true, 60% confidence” (implying 40% unknown, I think, not 40% false), but “80% likely to be true”. It really makes updates easier to calculate and understand.
The vast majority of yes/no questions you’re likely to face won’t support 5% intervals. You’re just not going to get enough data to have any idea whether the “true” calibration is what actually happens for that small selection of questions.
Tetlock found in the Good Judgement Project as described in his book Superforcasting that people who are excellent at forcasting do very finely grained predictions.
I disagree that you can’t get 5% intervals on random yes/no questions—if you stick with 10%, you really only have 5 possible values − 50-59%, 60-69%, 70-79%, 80-89%, and 90+%. That’s very coarse-grained.
The vast majority of yes/no questions you’re likely to face won’t support 5% intervals.
I agree [edit: actually, it depends on where these yes/no questions are coming from] , but think the questions I was looking at were in the small minority that do support 5% intervals.
Not “true, 60% confidence” (implying 40% unknown, I think, not 40% false)
Perhaps I should have provided more details to explain exactly what I did, because I actually did mean 60% true 40% false.
So, I already was thinking in the manner you advocate, but thanks for the advice anyway!
The vast majority of yes/no questions you’re likely to face won’t support 5% intervals. You’re just not going to get enough data to have any idea whether the “true” calibration is what actually happens for that small selection of questions.
That said, I agree there’s an analytic flaw if you can change true to false on no additional data (kind of: you noticed salience of something you’d previously ignored, which may count as evidence depending on how you arrived at your prior) and only reduce confidence a tiny amount.
One suggestion that may help: don’t separate your answer from your confidence confidence, just calculate a probability. Not “true, 60% confidence” (implying 40% unknown, I think, not 40% false), but “80% likely to be true”. It really makes updates easier to calculate and understand.
Tetlock found in the Good Judgement Project as described in his book Superforcasting that people who are excellent at forcasting do very finely grained predictions.
I disagree that you can’t get 5% intervals on random yes/no questions—if you stick with 10%, you really only have 5 possible values − 50-59%, 60-69%, 70-79%, 80-89%, and 90+%. That’s very coarse-grained.
I agree [edit: actually, it depends on where these yes/no questions are coming from] , but think the questions I was looking at were in the small minority that do support 5% intervals.
Perhaps I should have provided more details to explain exactly what I did, because I actually did mean 60% true 40% false.
So, I already was thinking in the manner you advocate, but thanks for the advice anyway!