The CRT is more standardly used for such measurement
I’m astonished by how many people attach such a great significance to that test. Three questions all of which are relatively common trick questions. For each question, I would expect about 60% of people to have already encountered it (or a very similar one) before somewhere else, about 20% of people who encounter it for the first time to give the correct answer, and about 90% of people who have encountered it before to remember (or remember how to derive) the correct answer.
In fact, I had thought that those three questions were only a sample, and when I realised they were the whole test, I was like ‘WTF?’
I think the researchers who use it were probably equally astonished at how much weight they end up putting on the results of the CRT, but it turns out to have massive predictive power—way more than you could possibly expect from a 3 question test. This is surprising, but that is not enough to reject it out of hand. Remember, if you think that a test which has been used repeatedly in a large body of scientific literature over many years has some obvious flaw or seems counter-intuitive for some reason, you should bear in mind that the scientists using it were probably at least as surprised as you are by the result, and will have made every effort to nail it down
As for the numbers you give, they cannot possibly be true. Even as late as 2011 (eg, in this paper) the mean number of correct answers given by participants was 0.7, and fewer than 5% of students got all three answers correct (your numbers would suggest a mean number of correct answers more than 1.5, and that more than 12% should get all three questions right, even if prior exposure to the three questions was independent).
Yes, it would be nice to have alternative tests to the CRT (I’d love to see a copy of the 8 question CRT Fredericks mentions briefly in one of his early papers on the topic), but the CRT happens to do its job pretty well. Intuitively, it seems as though it shouldn’t, but that’s why we don’t trust our intuition when we have data which contradict it.
Well, I must have overestimated those numbers (or maybe those kind of riddles are less popular where the study was performed than where I come from). Still, it might be interesting to also ask participants “Have you encountered this question (or a very similar question) before?” with possible answers “Yes, and I remember the answer (or how to derive it”, “Yes, but I have forgotten the answer”, “Yes, but I was never told the answer” and “No”.
I concur that the Ann/Bob/Carol question is more taxing than the Cognitive Reflection Test.
In fact I can prove for my own case sample size N = 1. I scored 3⁄3 on the CRT and I missed Ann/Bob/Carol as I did not look at Bob as being unambiguously either married or unmarried and shot myself in my own damn foot on the sucker.
I think I recall reading that when they actually give someone the CRT, they often mix those three questions in with a whole bunch of others… and just ignore all the other answers.
I’m astonished by how many people attach such a great significance to that test. Three questions all of which are relatively common trick questions. For each question, I would expect about 60% of people to have already encountered it (or a very similar one) before somewhere else, about 20% of people who encounter it for the first time to give the correct answer, and about 90% of people who have encountered it before to remember (or remember how to derive) the correct answer.
In fact, I had thought that those three questions were only a sample, and when I realised they were the whole test, I was like ‘WTF?’
I think the researchers who use it were probably equally astonished at how much weight they end up putting on the results of the CRT, but it turns out to have massive predictive power—way more than you could possibly expect from a 3 question test. This is surprising, but that is not enough to reject it out of hand. Remember, if you think that a test which has been used repeatedly in a large body of scientific literature over many years has some obvious flaw or seems counter-intuitive for some reason, you should bear in mind that the scientists using it were probably at least as surprised as you are by the result, and will have made every effort to nail it down
As for the numbers you give, they cannot possibly be true. Even as late as 2011 (eg, in this paper) the mean number of correct answers given by participants was 0.7, and fewer than 5% of students got all three answers correct (your numbers would suggest a mean number of correct answers more than 1.5, and that more than 12% should get all three questions right, even if prior exposure to the three questions was independent).
Yes, it would be nice to have alternative tests to the CRT (I’d love to see a copy of the 8 question CRT Fredericks mentions briefly in one of his early papers on the topic), but the CRT happens to do its job pretty well. Intuitively, it seems as though it shouldn’t, but that’s why we don’t trust our intuition when we have data which contradict it.
Well, I must have overestimated those numbers (or maybe those kind of riddles are less popular where the study was performed than where I come from). Still, it might be interesting to also ask participants “Have you encountered this question (or a very similar question) before?” with possible answers “Yes, and I remember the answer (or how to derive it”, “Yes, but I have forgotten the answer”, “Yes, but I was never told the answer” and “No”.
I concur that the Ann/Bob/Carol question is more taxing than the Cognitive Reflection Test.
In fact I can prove for my own case sample size N = 1. I scored 3⁄3 on the CRT and I missed Ann/Bob/Carol as I did not look at Bob as being unambiguously either married or unmarried and shot myself in my own damn foot on the sucker.
I think I recall reading that when they actually give someone the CRT, they often mix those three questions in with a whole bunch of others… and just ignore all the other answers.