I think the problem here is with many trivia questions you either know the answer or you don’t
That means that for those questions most probabilities are either close to 0 or close to 1. This suggests that given this set of questions it would probably be a good idea to increase “resolution” near those two points. For that purpose, perhaps instead of asking for confidence levels expressed as percentages you could ask for confidence levels expressed as odds or log odds. For example, users could express their confidence levels using odds expressed as ratios 2^n:1, for n=k,...,0,...,-k.
That’s an interesting thought but I do suspect that you’d have to answer a lot of questions to see any difference whatsoever. If you’re perfectly calibrated and answer 100 questions that you are either 99.99% confident or 99.9% confident, there’s a very good chance that you’ll get all 100 questions right, regardless of which confidence level you pick.
That means that for those questions most probabilities are either close to 0 or close to 1. This suggests that given this set of questions it would probably be a good idea to increase “resolution” near those two points. For that purpose, perhaps instead of asking for confidence levels expressed as percentages you could ask for confidence levels expressed as odds or log odds. For example, users could express their confidence levels using odds expressed as ratios 2^n:1, for n=k,...,0,...,-k.
That’s an interesting thought but I do suspect that you’d have to answer a lot of questions to see any difference whatsoever. If you’re perfectly calibrated and answer 100 questions that you are either 99.99% confident or 99.9% confident, there’s a very good chance that you’ll get all 100 questions right, regardless of which confidence level you pick.