My version would be that people decide “I gain n points if I get this right” for each answer they submit, where 0<=n<=10; if they get it wrong they lose 2^n points. Collapsing the multiple success criteria into a single score ensures miscalibration is penalized, but leaves no ambiguity in what to maximize.
Notes:
If scores update with each question, you might want to start players out with a small pool of points and then have them ‘go bust’ (i.e. get relegated to spectator and/or advisory roles) if their scores go negative. Otherwise, you’d end up with overconfident-and-unlucky people death-marching through the second half of the game with a score of −976 tied around their necks, which I imagine wouldn’t be fun.
My version still isn’t perfect because players will be torn between optimizing E(score) and P(I get the highest score): in a room full of people, the person who takes first place will probably be someone who was slightly more overconfident than the idealized EV-maxxing version of themselves. (This opportunity for circumrational reasoning could be seen as a feature but is honestly-probably-mostly a bug.)
Another imperfection is that there’s no clean separation between trivia skill and rationality skill. All I can say is that this was also true of the original; the Clever Fellows aren’t just demonstrating a blatant edge case, they’re showcasing a strategy which can be microdosed to smoothly and silently trade trivia score for rationality score; in addition to training calibration, your rules implicitly train “if you’re pretty sure your best guess is wrong, pretend you have no idea so you can at least appear better-calibrated”.
My version would be that people decide “I gain n points if I get this right” for each answer they submit, where 0<=n<=10; if they get it wrong they lose 2^n points. Collapsing the multiple success criteria into a single score ensures miscalibration is penalized, but leaves no ambiguity in what to maximize.
Notes:
If scores update with each question, you might want to start players out with a small pool of points and then have them ‘go bust’ (i.e. get relegated to spectator and/or advisory roles) if their scores go negative. Otherwise, you’d end up with overconfident-and-unlucky people death-marching through the second half of the game with a score of −976 tied around their necks, which I imagine wouldn’t be fun.
My version still isn’t perfect because players will be torn between optimizing E(score) and P(I get the highest score): in a room full of people, the person who takes first place will probably be someone who was slightly more overconfident than the idealized EV-maxxing version of themselves. (This opportunity for circumrational reasoning could be seen as a feature but is honestly-probably-mostly a bug.)
Another imperfection is that there’s no clean separation between trivia skill and rationality skill. All I can say is that this was also true of the original; the Clever Fellows aren’t just demonstrating a blatant edge case, they’re showcasing a strategy which can be microdosed to smoothly and silently trade trivia score for rationality score; in addition to training calibration, your rules implicitly train “if you’re pretty sure your best guess is wrong, pretend you have no idea so you can at least appear better-calibrated”.