If I’m reading the chart on that page correctly, Gwern is extremely well calibrated. Is the accuracy row for each confidence column telling us what fraction of predictions Gwern assigned a given confidence to have been right? He’s got 50% − 44%, 60% − 64%, 70% − 71%, 80% − 83%, 90% − 92%, and 100% − 96%. That’s incredible!
If I’m reading the chart on that page correctly, Gwern is extremely well calibrated. Is the accuracy row for each confidence column telling us what fraction of predictions Gwern assigned a given confidence to have been right? He’s got 50% − 44%, 60% − 64%, 70% − 71%, 80% − 83%, 90% − 92%, and 100% − 96%. That’s incredible!
Yes, something like that. I forget the exact details of how it bins.
Thank you. That’s years of practice and some useful heuristics at work there.