“across a broad spectrum of subject areas … questions are chosen randomly”
but this is the real weasel in there. Defining a good prior on “subject areas” is problematic. A very rational nerd would get wiped out if there are too many trivia questions… which is what happened to me just now on Tom’s rationality test:
Though my calibration on this test was very good, my Bayes Score was rubbish. Most of the questions were about America, (cultural bias) and most were about people (subject area bias). I like my idea of (calibration) * (Bayes Score).
“across a broad spectrum of subject areas … questions are chosen randomly”
but this is the real weasel in there. Defining a good prior on “subject areas” is problematic. A very rational nerd would get wiped out if there are too many trivia questions… which is what happened to me just now on Tom’s rationality test:
http://www.acceleratingfuture.com/tom/calibrate.php
Though my calibration on this test was very good, my Bayes Score was rubbish. Most of the questions were about America, (cultural bias) and most were about people (subject area bias). I like my idea of (calibration) * (Bayes Score).