The ‘test even if gamed’ reminds me of a labyrinth. Suppose there are several ways of reaching the end, and the participants can’t know which way they are set upon, because it is chosen randomly. They are asked questions from outside of their domain of knowledge (it would need a big database to pick from), constructed in such a way that it is impossible to pick the right answer without knowing about various cognitive biases (e.g., the conjunction fallacy etc.) The questions can be independently rated for apparent difficulty, and masters will be given the hardest ones.
(I don’t know what makes some questions seem simpler even if the answer is still wrong. I asked people to rate questions in my post ‘Before the seed. I. Guesswork’, and somehow people chose exactly one to be ‘the easiest to formulate hypotheses about’, but I don’t know how they did it. Plus, few people answered at all.)
The ‘test even if gamed’ reminds me of a labyrinth. Suppose there are several ways of reaching the end, and the participants can’t know which way they are set upon, because it is chosen randomly. They are asked questions from outside of their domain of knowledge (it would need a big database to pick from), constructed in such a way that it is impossible to pick the right answer without knowing about various cognitive biases (e.g., the conjunction fallacy etc.) The questions can be independently rated for apparent difficulty, and masters will be given the hardest ones. (I don’t know what makes some questions seem simpler even if the answer is still wrong. I asked people to rate questions in my post ‘Before the seed. I. Guesswork’, and somehow people chose exactly one to be ‘the easiest to formulate hypotheses about’, but I don’t know how they did it. Plus, few people answered at all.)