I like this. One suggestion: give students a small, predefined set of credences they’re allowed to use, so they don’t have to spend time deciding between e.g. 33% and 34%.
I’d suggest: 10%, 25%, 50%, 75%, and 90%, or maybe just all the deciles.
(Note: I believe it’s not critical w/ the given scoring rule that the probabilities sum exactly to 100%. So you might end up guessing 10, 10, 25, 50, and that’s okay. Someone please correct me if this is wrong.)
I’m unclear on the benefits. I can see how it’s sometimes faster, but I’m unclear why faster is an important criterion, and it will sometimes be slower: if I quickly generated the first example in the post of 33,33,33,1, then I’d likely slow down a good deal trying to decide which of those 33s to turn into a 30 and which to turn into a 40.
Whereas it seems clearly valuable to encourage a habit of having the probabilities sum to 100%. That’s not for the scoring rule, it’s for developing good intuitions.
if I quickly generated the first example in the post of 33,33,33,1
If you’re doing a test with a bunch of these questions, and you have the same probabilities to choose from every time, then I’d expect you to quickly generate those numbers instead, and not have this problem.
I like this. One suggestion: give students a small, predefined set of credences they’re allowed to use, so they don’t have to spend time deciding between e.g. 33% and 34%.
I’d suggest: 10%, 25%, 50%, 75%, and 90%, or maybe just all the deciles.
(Note: I believe it’s not critical w/ the given scoring rule that the probabilities sum exactly to 100%. So you might end up guessing 10, 10, 25, 50, and that’s okay. Someone please correct me if this is wrong.)
I’m unclear on the benefits. I can see how it’s sometimes faster, but I’m unclear why faster is an important criterion, and it will sometimes be slower: if I quickly generated the first example in the post of 33,33,33,1, then I’d likely slow down a good deal trying to decide which of those 33s to turn into a 30 and which to turn into a 40.
Whereas it seems clearly valuable to encourage a habit of having the probabilities sum to 100%. That’s not for the scoring rule, it’s for developing good intuitions.
If you’re doing a test with a bunch of these questions, and you have the same probabilities to choose from every time, then I’d expect you to quickly generate those numbers instead, and not have this problem.
You could normalize the scoring rule back to 1, so that should be fine.
Probabilities could be done with fractions (though there’s the risk of error). Odds could also be given.