Part of the output of your quizzes is a line of the form “Your chance of being well calibrated, relative to the null hypothesis, is 50.445538580926 percent.” How is this number computed?
I chose “25% confident” for 25 questions and got 6 of them (24%) right. That seems like a pretty good calibration … but 50.44% chance of being well calibrated relative to null doesn’t seem that good. Does that sentence mean that an observer, given my test results, would assign a 50.44% probability to my being well calibrated and a 49.56% probability to my not being well calibrated? (or to my randomly choosing answers?) Or something else?
It’s also completely ridiculous, with a sample size of ~10 questions, to give the success rate and probability of being well calibrated as percentages with 12 decimals. Since the uncertainty in such a small sample is on the order of several percent, just round to the nearest percentage.
Part of the output of your quizzes is a line of the form “Your chance of being well calibrated, relative to the null hypothesis, is 50.445538580926 percent.” How is this number computed?
I chose “25% confident” for 25 questions and got 6 of them (24%) right. That seems like a pretty good calibration … but 50.44% chance of being well calibrated relative to null doesn’t seem that good. Does that sentence mean that an observer, given my test results, would assign a 50.44% probability to my being well calibrated and a 49.56% probability to my not being well calibrated? (or to my randomly choosing answers?) Or something else?
It’s also completely ridiculous, with a sample size of ~10 questions, to give the success rate and probability of being well calibrated as percentages with 12 decimals. Since the uncertainty in such a small sample is on the order of several percent, just round to the nearest percentage.
It probably just computes it as a float and then prints the whole float.
(I do recognize the silliness of replying to a three-year old comment that itself is replying to a six-year old comment.)
It’s not silly. I still find these newer comments useful.
And here we are one year later!
Yes, do it for posterity!
I would like to chime in and point out that as today the domain “acceleratingfuture (dot) com” is owned by a russian bookmaker.