I Made a Judgment Calibration Game for Beginners (Calibrate)
I made a game that teaches beginner calibration. Have you ever wanted your brother/girlfriend/aunt/dog to make more calibrated probabilistic claims? Now you can send them Calibrate! (Or you can play it yourself.)
Play now (desktop only)
Semi-big catch: Calibrate was mostly made for computer science students, so non-computer science students may have a hard time getting very high scores. People from other hard sciences have also enjoyed it, though!
More on Calibrate
The premise of the game is to get rich by investing in a calibrated way. It’s more fun than existing calibration training tools in my opinion, because there’s more sense of progress from earning money and moving through the game. It’s not very long, roughly 30 minutes total.
I also did a very low-powered analysis of calibration outcomes with 11 play testers. The results look promising; see the calibration curve on the left. See the plot on the right for how to interpret the curve. On a cursory look, outcomes are looking similar to those of other calibration trainings, which are usually much longer. (Confidence interval calibration didn’t work super well compared to multiple-choice confidence calibration; see the unhappy green X on the left.)
Other excellent calibration tools (not computer games):
Thanks for making the game! I played it, and was a little frustrated that not all of your questions had the correct answers. For example, I recall one of the questions as reading, “How many typical transport protocols are on the internet?” which is not quite the same as what was answered, “How many internet transport protocols are commonly used?” You see, any protocol is “typical” for its use cases, even if it is uncommon. It reminded me somewhat of taking the SAT, where the test writers would replace a word with a more common word which doesn’t actually quite mean what they intended. As the commenter below me pointed out, what he does with such questions is go meta—and I kind of like that perspective—but it was nevertheless a little frustrating. Overall though, I liked the game. Thanks for making it.
Note: I edited this the day after I wrote it because I found it a little too antagonistic, and I don’t want to be antagonistic. I really did like the game, and appreciate that you made it.
One thing I do with such questions is go meta, i.e. add some uncertainty for P(the tool actually has the right answer)when giving my responses.