If a person was perfectly calibrated, then each 10% increase in their probability estimate would translate into a 10% higher likelihood of getting the answer correct. If you plot probability estimates on the x axis and whether or not the event happened on the y axis, then you should get a slope of 1 (the line y=x). But people tend to be miscalibrated—out of the questions where they say “90%”, they might only get 70% correct. This results in a shallower slope (in this example, the line would go through the point (90,70) instead of (90,90)) - a slope less than 1.
I took the 1141 people’s answers to the 10 calibration questions as 11410 data points, plotted them on an x-y graph (with the probability estimate as the x value and a y value of 100 if it’s correct and 0 if it’s incorrect), and ran an ordinary linear regression to find the slope of the line fit to all 11410 data points.
That line had a slope of 0.91. In other words, if a LWer gave a probability estimate that was 10 percentage points higher, then on average the claim was 9.1 percentage points more likely to be true. Not perfect calibration, but not bad.
If we look at various subsets of LWers on the survey, here are the slopes that we get:
0.91 Everyone 0.92 Read HPMOR 0.92 1000 karma 0.93 High test scores 0.93 Read the sequences 0.96 Active in-person 0.96 Attended CFAR
I haven’t done any tests of statistical significance, but all of these more LWy subgroups do have slopes that are higher (and closer to the well-calibrated slope of 1) than the slope for the full sample (as do the people with high scores on SAT/ACT/IQ tests).
And here’s an analysis of calibration.
If a person was perfectly calibrated, then each 10% increase in their probability estimate would translate into a 10% higher likelihood of getting the answer correct. If you plot probability estimates on the x axis and whether or not the event happened on the y axis, then you should get a slope of 1 (the line y=x). But people tend to be miscalibrated—out of the questions where they say “90%”, they might only get 70% correct. This results in a shallower slope (in this example, the line would go through the point (90,70) instead of (90,90)) - a slope less than 1.
I took the 1141 people’s answers to the 10 calibration questions as 11410 data points, plotted them on an x-y graph (with the probability estimate as the x value and a y value of 100 if it’s correct and 0 if it’s incorrect), and ran an ordinary linear regression to find the slope of the line fit to all 11410 data points.
That line had a slope of 0.91. In other words, if a LWer gave a probability estimate that was 10 percentage points higher, then on average the claim was 9.1 percentage points more likely to be true. Not perfect calibration, but not bad.
If we look at various subsets of LWers on the survey, here are the slopes that we get:
0.91 Everyone
0.92 Read HPMOR
0.92 1000 karma
0.93 High test scores
0.93 Read the sequences
0.96 Active in-person
0.96 Attended CFAR
I haven’t done any tests of statistical significance, but all of these more LWy subgroups do have slopes that are higher (and closer to the well-calibrated slope of 1) than the slope for the full sample (as do the people with high scores on SAT/ACT/IQ tests).