These examples emphasize the benefit of frequently taking calibration tests, where we assign probabilities to answers and then checks those answer for calibration errors. Perhaps someone could create a website where we could do this regularly? Just collect a large list of questions like the ones above, questions with true answers but where we have intuitions about what the answer might be, and then have us answer those questions with probabilities, and then show us a calibration chart for the last X questions. Yes, collecting the good questions will be most of the work.
What if I were to try to create such a web app. Should I take 5 minutes every lunchbreak asking friends and colleagues to brainstorm for questions? Maybe write a LW post asking for questions? Maybe there could be a section of the site dedicated to collecting and curating good questions (crowdsourced or centrally moderated).
These examples emphasize the benefit of frequently taking calibration tests, where we assign probabilities to answers and then checks those answer for calibration errors. Perhaps someone could create a website where we could do this regularly? Just collect a large list of questions like the ones above, questions with true answers but where we have intuitions about what the answer might be, and then have us answer those questions with probabilities, and then show us a calibration chart for the last X questions. Yes, collecting the good questions will be most of the work.
What if I were to try to create such a web app. Should I take 5 minutes every lunchbreak asking friends and colleagues to brainstorm for questions? Maybe write a LW post asking for questions? Maybe there could be a section of the site dedicated to collecting and curating good questions (crowdsourced or centrally moderated).
CFAR has 2 apps you might find interesting; I was able to find them on apple store easily. http://rationality.org/apps/
Are those apps only available on Apple products/smartphones? No way to access them on a Windows PC?
The Credence Calibration game is also available for Windows. Links to download it in the various formats are here.
The calibration game is also available for Android and was available for Windows but I think the original website is down.