I think the researchers who use it were probably equally astonished at how much weight they end up putting on the results of the CRT, but it turns out to have massive predictive power—way more than you could possibly expect from a 3 question test. This is surprising, but that is not enough to reject it out of hand. Remember, if you think that a test which has been used repeatedly in a large body of scientific literature over many years has some obvious flaw or seems counter-intuitive for some reason, you should bear in mind that the scientists using it were probably at least as surprised as you are by the result, and will have made every effort to nail it down
As for the numbers you give, they cannot possibly be true. Even as late as 2011 (eg, in this paper) the mean number of correct answers given by participants was 0.7, and fewer than 5% of students got all three answers correct (your numbers would suggest a mean number of correct answers more than 1.5, and that more than 12% should get all three questions right, even if prior exposure to the three questions was independent).
Yes, it would be nice to have alternative tests to the CRT (I’d love to see a copy of the 8 question CRT Fredericks mentions briefly in one of his early papers on the topic), but the CRT happens to do its job pretty well. Intuitively, it seems as though it shouldn’t, but that’s why we don’t trust our intuition when we have data which contradict it.
Well, I must have overestimated those numbers (or maybe those kind of riddles are less popular where the study was performed than where I come from). Still, it might be interesting to also ask participants “Have you encountered this question (or a very similar question) before?” with possible answers “Yes, and I remember the answer (or how to derive it”, “Yes, but I have forgotten the answer”, “Yes, but I was never told the answer” and “No”.
I think the researchers who use it were probably equally astonished at how much weight they end up putting on the results of the CRT, but it turns out to have massive predictive power—way more than you could possibly expect from a 3 question test. This is surprising, but that is not enough to reject it out of hand. Remember, if you think that a test which has been used repeatedly in a large body of scientific literature over many years has some obvious flaw or seems counter-intuitive for some reason, you should bear in mind that the scientists using it were probably at least as surprised as you are by the result, and will have made every effort to nail it down
As for the numbers you give, they cannot possibly be true. Even as late as 2011 (eg, in this paper) the mean number of correct answers given by participants was 0.7, and fewer than 5% of students got all three answers correct (your numbers would suggest a mean number of correct answers more than 1.5, and that more than 12% should get all three questions right, even if prior exposure to the three questions was independent).
Yes, it would be nice to have alternative tests to the CRT (I’d love to see a copy of the 8 question CRT Fredericks mentions briefly in one of his early papers on the topic), but the CRT happens to do its job pretty well. Intuitively, it seems as though it shouldn’t, but that’s why we don’t trust our intuition when we have data which contradict it.
Well, I must have overestimated those numbers (or maybe those kind of riddles are less popular where the study was performed than where I come from). Still, it might be interesting to also ask participants “Have you encountered this question (or a very similar question) before?” with possible answers “Yes, and I remember the answer (or how to derive it”, “Yes, but I have forgotten the answer”, “Yes, but I was never told the answer” and “No”.