I think that there are better analyses of calibration which could be done than the ones that are posted here.
For example, I think it’s better to combine all 10 questions into a single graph rather than looking at each one separately.
The pattern of overconfidence on hard questions and underconfidence on easy questions is actually what you’d expect to find, even if people are well-calibrated. One thing that makes a question easy is if the obvious guess is the correct answer (like a question about Confederate Civil War generals where the correct answer is Robert E. Lee). On those sorts of easy questions, a bunch of people who made the obvious guess with moderate confidence will turn out to be correct, and they’ll look underconfident. Whereas on questions where the correct answer is more obscure or counterintuitive, those guesses will be wrong and they’ll look overconfident.
I’ll see what other analyses I can do with the data. First I’ll need to make some not-entirely-straightforward decisions about what to do with misspellings and other interesting responses. This may be fairly important, since those account for a decent chunk of all responses. For example, here are the most common responses to question 8 (in descending order of frequency):
I would count minor typos (like ‘mitochondira’) and spelling errors (like ‘mitocondria’), and trivial variants (like ‘mitochondrium’) as correct. I’d count major typos, where the pronunciation would be majorly different from the correct name, as neither correct nor incorrect—ditch the data, since it may require too many judgment calls about whether someone’s zeroing in on the correct name, and since the respondents themselves weren’t told how much to make their calibration correct-spelling-sensitive. So my system says:
Heisenberg
Also correct: Werner Heisenberg, Heisenburg, Heissenburg
Incorrect: Heinsperbg
Neither: Herzinberg
Hawai’i
Also correct: Hawai`i, Hawaii, Hawii, Hawai, HI
Neither: HA, Hawaii or Arkansas
Odin
Also correct: Oden, Wudin, Wodin, Wodan, Wotan, Oðinn, Woden
Incorrect: Oris, Alduin
Jacob
Also correct: Yaakov, Jacob aka. Israel, Israel, Jocob, Jakob, Jacob/Israel, Jacob aka Israel
I think that there are better analyses of calibration which could be done than the ones that are posted here.
For example, I think it’s better to combine all 10 questions into a single graph rather than looking at each one separately.
The pattern of overconfidence on hard questions and underconfidence on easy questions is actually what you’d expect to find, even if people are well-calibrated. One thing that makes a question easy is if the obvious guess is the correct answer (like a question about Confederate Civil War generals where the correct answer is Robert E. Lee). On those sorts of easy questions, a bunch of people who made the obvious guess with moderate confidence will turn out to be correct, and they’ll look underconfident. Whereas on questions where the correct answer is more obscure or counterintuitive, those guesses will be wrong and they’ll look overconfident.
I’ll see what other analyses I can do with the data. First I’ll need to make some not-entirely-straightforward decisions about what to do with misspellings and other interesting responses. This may be fairly important, since those account for a decent chunk of all responses. For example, here are the most common responses to question 8 (in descending order of frequency):
Mitochondria
mitochondria
Mitochondrion
mitochondrion
Nucleus
Mitocondria
mitochondrium
nucleus
mitocondria
Mitochondrium
Mithocondria
Mitochondira
Ribosome
I would count minor typos (like ‘mitochondira’) and spelling errors (like ‘mitocondria’), and trivial variants (like ‘mitochondrium’) as correct. I’d count major typos, where the pronunciation would be majorly different from the correct name, as neither correct nor incorrect—ditch the data, since it may require too many judgment calls about whether someone’s zeroing in on the correct name, and since the respondents themselves weren’t told how much to make their calibration correct-spelling-sensitive. So my system says:
Heisenberg
Hawai’i
Odin
Jacob
Indonesia
Hawaii is both in the ‘Also correct’ and the ‘Neither’ list.
One uses a typewriter apostrophe (’), the other doesn’t (`).
No, bbleeker is saying that “
Hawaii
” (no apostrophe) is in both lists.Ah. “Hawaii or Arkansas” is its own entry, and was typed in as an answer.