“Piggyback” on other tests: ask people taking part in various tests (standardized exams, sport competitions, driving lessons, programming contests, art exhibitions—whatever) their chances of success (or their probability distribution over the range of results).
The other items should themselves be important enough, so it would fit well with a university cursus, so that it can be “automated” for a lot of things. The way of asking for predictions should be made so as to maximize bad predictions: for example the students are asked to give estimations in front of their peers (if that’s shown to get them to overestimate), but afterwards not reminded of the prediction they gave nor of whether it came true (so that they don’t deliberately try to make it come true).
It could also be extended to other events like “when I’ll turn in my thesis” or even “whether I’ll be single in a year” or “how much I’ll weight in six months”.
The more subjects they have to estimate on, the better. At the end, measure the Bayes-score.
This could be combined to some more “dramatic” and explicit rationality tests (see the other comments) to constitute the scoring method of a university rationality course. The explicit rationality tests would also help take a bit of attention away from the day-to-day probability estimates on exams and stuff, to diminish the “only rational when deliberately thinking about it” phenomenon.
Oh, also—ask the students for an estimate before the exam and after the exam (but before they have a chance of talking to someone else). Maybe even a week before and a week after too.
“Piggyback” on other tests: ask people taking part in various tests (standardized exams, sport competitions, driving lessons, programming contests, art exhibitions—whatever) their chances of success (or their probability distribution over the range of results).
The other items should themselves be important enough, so it would fit well with a university cursus, so that it can be “automated” for a lot of things. The way of asking for predictions should be made so as to maximize bad predictions: for example the students are asked to give estimations in front of their peers (if that’s shown to get them to overestimate), but afterwards not reminded of the prediction they gave nor of whether it came true (so that they don’t deliberately try to make it come true).
It could also be extended to other events like “when I’ll turn in my thesis” or even “whether I’ll be single in a year” or “how much I’ll weight in six months”.
The more subjects they have to estimate on, the better. At the end, measure the Bayes-score.
This could be combined to some more “dramatic” and explicit rationality tests (see the other comments) to constitute the scoring method of a university rationality course. The explicit rationality tests would also help take a bit of attention away from the day-to-day probability estimates on exams and stuff, to diminish the “only rational when deliberately thinking about it” phenomenon.
Oh, also—ask the students for an estimate before the exam and after the exam (but before they have a chance of talking to someone else). Maybe even a week before and a week after too.