For a much less gameable and more fun version of Calibration Trivia, use the rules that the bar I play trivia at uses.
The game is broken up into rounds. Within a round, players/teams have 2, 4, 6, 8, and 10 points that they can assign to their answer as they submit it. They can use each only once. Here is the sequence of events:
Read out the 5 categories for the 5 questions making up the round. This way folks know roughly how to allocate their points.
Read the first question. Folks must submit their answers and the number of points they will score if their answer is correct. These come in simultaneously on the same sheet. At this time they know the categories of the other 4 questions but don’t know the questions, so they will have to estimate their knowledge of the topic areas of questions 2-5, and compare that to their confidence about their answer to question 1.
Then do the 2nd, 3rd, 4th, and 5th questions the same way. Each point value can be used only once, so the maximum score is 30 points—but you can score up to 18 points even knowing only 2⁄5 answers if you are correctly calibrated.
You can adapt this basic structure to test calibration more or less. You could, e.g., read all 5 questions and have folks assign the point values to all 5 answers before turning any of them in, or require assigning point values to the categories before hearing any of the questions. You could make the points a wager (allowing negative scores) - but allow people to bet zero some number of times. You could increase/decrease/change the point values per round.
But this basic structure allows you to reward good calibration without rewarding stupid calibration. Knowing some answers in this game is necessary but not sufficient to win.
Yeah! The time between the invention of the bound book and the invention of the printing press was about 1400 years. In the intervening time, books were very expensive.