Ask LW: What questions to test in our rationality questionnaire?
We’ve had quite a bit of discussion around LW, and OB, on the questions:
Is there a robust trait, “rationality”, that predicts accurate belief-formation in humans?
If so, how can we measure it? And what kinds of training might help?
Also, does “rationality” in the above sense help people achieve other goals, such as income, happiness, personal growth, positive relationships, or world-saving?
Rationalists that we are, it’s time to put our experiments where our mouths are. So here’s my plan:
Step 1: Assemble a set of questions that might possibly help us understand: (a) how rational people are; (b) where they got that rationality from; and (c) what effects their rationality has on their lives. Include any questions that might help in the formulation of useful conjectures. After collecting the data, look for correlations, spaghetti-at-the-wall style. Try factor analysis.
Step 2 [Perhaps after iterating the quick-and-dirty Step 1 correlational approach a bit, to develop better candidate metrics]: Run some more careful experimental tests of various sorts, both with a “rationality training group” that meets for extended periods of time, and, if LW is willing, with shorter training experiments with randomized LW subgroups. Try to build an atmosphere and knowledge base on LW where more people go out and do useful experiments.
I have an initial questionnaire draft below, although I skipped the answer-choices for brevity. Please post your suggestions for informative questions include and/or to drop. As good suggestions come in, I’ll edit the questionnaire draft to include them. It would be nice if the questionnaire we actually use draws on the combined background of the LW community.
Please also post hypotheses for what kinds of correlations you expect to see and/or to not see, when the questionnaire is actually run. If you note your hypotheses now, before the data comes in, we’ll know we should increase our credence in your theory instead of just accusing you of hindsight bias.
Once we have a good questionnaire draft, I’ll put the questionnaire on the web and call for LW readers to fill out the questionnaire. I’ll also try to get people to fill out the questionnaire from some non-LW groups, e.g. Stanford students. Then I’ll post the questionnaire data, and we can all have fun interpreting it.
Section A. Demographic information. Possible confounders, i.e. variables other than “rationality” that may influence correct beliefs.
Age (from a multiple choice list, so we don’t identify individuals)
Sex [Why: everyone else asks for these, and they might have good reason.
SAT, ACT, and GRE scores, if any. [Why: as a proxy for IQ. IQ helps with many cognitive tasks, probably including rationality questions. We want to be able to tell the difference between “IQ helps people earn money” and “rationality helps people earn money, even after controlling for IQ”.]
Section B. Educational variables that may help cause rationality.
Parents’ education.
Parents’ scientific literacy.
Parents’ religious views.
Whether your parents were crazier than average, and/or more rational than average.
Amount of formal education. College major.
Occupation.
How many non-fiction books did you read in the last month? How many fiction books? [Why: people are probably more likely to give accurate data if we ask about e.g. “the last month”, than if we ask vaguer question like “how much do you usually read?”]
How many self-help or business books did you read in the last month?
When is the last time you sought out someone who was better than you at some skill you wanted to learn, and you asked them questions to try to figure out what you should be doing?
Have you read any books about heuristics and biases?
Have you read OB or LW at all?
If yes: 11a. When did you start reading?
11b. What portion have you read?
11c. Do you discuss the ideas with anyone, either online (e.g., as a commenter), or in person?
Which of the following activities have you trained in: mathematics, programming, engineering or practical tinkering, music, meditation, martial arts, debate, strategy games (go, chess, backgammon, etc.).
Section C. Indicators of real-world success.
Income.
[Marriage and divorce history? Whether you’re in a stable relationship? Whether you’re happy with their relationship? Whether you have an easy time getting dates? How do people usually test for “success” here?]
Number of best friends, for some operationalizations of “best friends” (e.g., people you could borrow $500 from; people with whom you can talk about nearly anything; ?) [What questions are standard, here?]
Whether you’ve ever been in a car accident
Happiness
Whether you’ve been overall “more successful”, “less successful”, or “about as successful” as most people in their high school graduating class, and in their college graduating class.
Whether you’ve “learned more”, “learned less”, or “learned about as much” since graduating {high school / college} as most people in your {high school / college} graduating class.
How often did you exercise in the last week?
Do you smoke?
High school and college GPAs
Do you have a current driver’s license?
Are there any late bills, bounced checks, bad debts, etc. on your credit record?
How many dental cavities did you get in the last two years?
Section D. Standard heuristics and biases questions
[Several standard questions, and variations on standard questions, that I’d rather not give details on so I don’t cause LW readers to get them right. The goal here is to find ways of testing for standard biases among people who have read the standard articles. If anyone has clever ideas for how to disguise the questions, please do email your ideas to annasalamon at gmail, and please don’t post your ideas in the comments.]
Section E. Current beliefs
Religious views.
Are you signed up for cryonics? Views on cryonics.
Views on group (gender, race) differences in IQ. (Not the origins of the differences; just whether there are group differences in today’s adults).
Views on the odds nuclear war over the next few decades
How good-looking are you, relative to other people of your age and gender?
Views on Pascal’s wager
Views on consciousness
Views on evolution
Views on whether global warming is happening, and whether it is significant
[Why: to see how good people are at forming accurate beliefs. And to get a bit of information on whether the above beliefs are accurate, by seeing whether the beliefs correlate with other rationality-indicators.]
Section F. Value placed on truth
Is it better to have accurate beliefs, or beliefs that give you morale or meaning?
Do you try to believe good things about your friends?
Do you try to believe good things about people who are different from you (e.g., people of different ethnic or religious backgrounds, people from different countries, people with different sexual orientations)? Why?
How important is it to you to have accurate beliefs?
Imagine a scale from 1 to 10 that measures the process by which you form beliefs. Let “1” mean “there may be emotional or other non-rational pressures, but those pressures have little impact on my resulting conclusions”. Let “10” mean “I basically just made up these beliefs because they felt good, seemed socially useful, matched my fears, or had some other non-truth-related property”. On this scale, how did you form your beliefs concerning:
a. Yourself
b. Your friends and family
c. How to make money, gain skills, make friends, etc.
d. The larger world (e.g., how likely the economy is to do well in the next few years, whether nuclear weapons or global warming pose real risks, what impact different political parties might have, etc.).
Section G. Attempts to seek information
In the last week, how much time did you spend trying to understand:
a. Yourself
b. Your friends and family
c. How to make money, gain skills, make friends, etc.
d. The larger world (e.g., how likely the economy is to do well in the next few years, whether nuclear weapons or global warming pose real risks, what impact different political parties might have, etc.).
How well do you know your friends and family? How do you know how well you know them?
How well do you know yourself? How do you know?
How well do you understand those aspects of the world that enable real-world measurable success, e.g. income? How do you know?
Have you experimented with different ways to do your job effectively?
Have well do you understand the larger world? How do you know?
Think about the last time you had a fight or conflict with someone. How much time did you spend rehearsing the evidence for your side? How much time did you spend trying with honest curiosity to figure out what happened?
How often do you notice that one of your pieces of knowledge conflicts with your model of some other part of the world (e.g., that you don’t understand why the floating toy in the pool bops to the top at the angle it does, or why
Section H: Models of one’s own thinking skill [This is the only section with open-ended rather than multiple-choice questions. Respondants can skip this section while filling out the rest]
What’s the worst mistake you made in the last year? What did you do about it?
What are the largest gaps in your current thinking skills?
What are your greatest strengths as a thinker?
On what topics are you most prone to self-deception?
What is the biggest improvement you’ve made in your ability to form accurate beliefs over the last year?
What safeguards do you use, to try to notice flaws in your own beliefs?
ADDED: The idea here is not to generate an actual, first-round test of individuals’ rationality. The idea is to take a bunch of questions that might plausibly correlate with that nebulous mix of concepts, “rationality”, and to see how well those questions correlate with one another. We won’t get a “your’re more rational than 70% of the population” out of this questionnaire: no way, no how. We may well get a some suggestive data about clusters of questions and answers where respondants’ answers tend to correlate with one another, and so suggest possible underlying factors worth more careful investigation.
Psychologists often do cheap, bad studies before they do slow, careful, expensive studies, to get an initial look at what might be true.
“How many non-fiction books did you read in the last month? How many fiction books?”
I would rather phrase this: “How many pages have you read in the last month?” I’ve read zero books, but I’ve read ~200 pages of journal articles. Others may read hundreds of newspaper pages in a month. Plus, some books are 200 pages long, others are 600 pages long, so it’s better to ask directly about pages of text, irrespective of the medium.
“Are you signed up for cryonics? Views on cryonics.”
This is a LW / OB bias. Why cryonics as opposed to the many other technoscientific and transhumanist topics out there? Because Eliezer and Robin hyped it up.
Well, there’s not a lot you can sign up for, transhumanist-wise. What might be a supplement/replacement for this question?
Do you exercise regularly? Do you use prediction markets/participate in them? Do you invest in index/ETF funds instead of mutual funds? I’m drawing a blank on these sorts of practical things.
Why don’t you plan to leverage the multitude of quality psychological tests?
The first two and most obvious are a personality and an IQ test. For personality, psychologists usually use the big five factor test, which can be found online with a well-calibrated question set e.g. here: http://www.personalitytest.net/ipip/ipipneo1.htm
This is a great idea. But… I think you have to look towards your cultural bias. I am retired after a full career and have a Bachelor Degree. I have no idea what I am supposed to do with questions about SAT, ACT, GRE, GPA, college majors. Is someone who was educated partly in Canada, partly in the UK (and not recently) not welcome? How would you think someone who is from a non-English speaking region and who has English as a second language would view the questionnaire? At least “IQ” and “highest academic level achieved” are somewhat universal ideas. Good luck.
This sounds like a huge project to put together from scratch. I recommend looking into some of the existing research on individual differences in rationality, such as Keith Stanovich’s books (Who Is Rational? and What Intelligence Tests Miss: The Psychology of Rational Thought seem most relevant) and papers (Stanovich & West 1997 looks particularly useful—they measure people’s sensitivity to argument quality on a range of real-life issues).
What do you mean by significant? How did you choose the questions under E? It’s probably best to be explicit to avoid suspicions of trying to set the test up to reach some particular conclusion. (For example, if you choose questions on which Blues are irrational, the test will find rationality to anti-correlate with giving Blue answers.)
What is this test for, exactly?
I’m much more interested in techniques by which I can measure my own rationality and discover whether it is improving, than the means by which I can measure other people’s. After all, if my own rationality is good then my estimates of other people’s rationality should directly improve.
And I have very strong misgivings about the idea of trying to measure it by asking intensely personal questions that may or may not correlate with rationality. It’s not at all clear to me that a rational person would choose to answer, or would apply to join an organisation that asked in the first place.
If the idea is to set a MENSA-like membership gate based on rationality, I would opt for directly testing only for what we can uncontroversially test for, such as basic familiarity with the heuristics and biases program, and asking in addition that they state their agreement with the broad outlines of what brings us together, such as the belief that the territory is not just another map.
ciphergoth, see my “ADDED” at the end of the post.
The idea here isn’t to tell individuals how rational they are, and it certainly isn’t to set a gate on other people. It’s… to feel around through the vague space of notions of “rational” that seem to be in use, and see if there might be a notion in there that has some construct validity.
I think the only real use of such a survey will be to more precisely pin down how members of this community are using the term “rationality.” And by the construction of the survey, it seems pretty clear that what they (or you) mean by “rational” is just “whatever we are.”
OK, it’s not meant as a test yourself or a membership test, but a sort of psychological experiment. I have to say that I don’t have a feel for the query you’re trying to hug—for example, do you have an application of this information in mind?
Later we can do a test that will determine how “rational” you are more or less, the problem is we don’t really have a good experimental definition of rationality. This survey will help see correlations and underlying characteristics of rational people (or at least people striving for rationality).
We don’t really know what to look for right now, so a broad set of questions will help us find out what is worth looking at in more depth. Which also means we can’t really know what we will get out of this, but hopefully we can find some strong indicator type questions for rational action.
Science is not about rushing off to collect data before you have any idea why you want it or what you’re going to do with it! If you have no idea what you might even hope to get out of an exercise in collecting data, what reason do you have to hope that good will come of it? What use would you make of “strong indicator type questions for rational action”? What is the underlying query?
None of these questions directly measure “rationality” in any real way. The most you’re going to glean from such data will be a demographic slice of the readership of LW/OCB. Now, I imagine this group will be somewhat more “rational” than average (if only due to higher “g,” and the presumed correlation between the two; I wouldn’t even be so quick to ascribe greater rationality to people professing an interest in the concept), but it’s patently silly to equate “being a reader of LW/OCB” with “possessing the quality of rationality.”
I’m very glad to see you doing this. I was going to try something similar (see the discussion here ) but I got lazy and distracted and I probably wasn’t up to the data processing tasks anyway.
I was throwing together a questionnaire here. You’re welcome to look over it and take any questions you think are interesting.
The survey’s goals were different from yours—it was more of an attempt to collect information on Less Wrong members—but enough of the questions look similar to yours that you might be interested in seeing the rest.
Most questions benefit from an “other” field—there’s always an awkward bugger that doesn’t quite fit. There are quite a few of your questions I’d like to choose the “other” answer on.
If some mad hacker adds a polls feature to the LessWrong codebase (which could be fun), they should have the code mandate an other answer on all polls...
“I had a professor...who taught the introduction to tactical engineering course. He said he never bothered changing his tests from term to term to prevent cheating, because while the questions were always the same, the answers changed. I’d thought he was joking.” --Miles Vorkosigan, Memory, Lois McMaster Bujold
Rationality isn’t a thing, it’s a state. Rational is not a property, it’s a conclusion.
If you want to test for rationality, ask questions that require rationality to get the right answer.
Any suggestions? That’s basically the idea with section D (the heuristics and biases type questions, that have correct answers) and (with more interpretive ambiguity, because it is less obvious which beliefs are correct) with section E (questions about current beliefs).
Set up questions that require you assume something odd in the preamble, and then conclude with something unpalatable (and quite possibly false). This tests to see if people can apply rationality even when it goes against their emotional involvement and current beliefs. As well as checking that they reach the conclusion demanded (logic), also give them an opportunity as part of a later question to flag up the premise that they feel caused the odd conclusion.
Something bayesian—like the medical test questions where the incidence in the general population is really low, but that specific one has been done so much loads of people know it. Maybe take some stats from newspaper reports and see if appropriate conclusions can be drawn.
“When was the last time you changed your mind about something you believed?” tests peoples ability to apply their rationality.
I could make all sorts of suggestions. But I’d rather straighten our your understanding so that you can produce as many examples of rationality as you like.
Teach you to fish rather than just giving you one, so to speak.
What’s rationality?
When we find components, how will we know which one to call “rationality”? If the point is that it “predicts accurate belief-formation”, doesn’t the test interpreter need to know more about what beliefs are accurate than the average test-taker? We might have preconceptions as to what answers accurate belief-formers may give on questions not about beliefs, but it seems to me these aren’t very helpful unless they’re very strong, and my preconceptions on this aren’t very strong.
Section D will have well-defined right answers. Some will even be unknown to test-takers and long-time LW readers, e.g. we can ask for confidence intervals on unfamiliar trivia and we can see if the real answers to the trivia problems fall within questioners’ 99% confidence intervals or not.
You’re right about section E as far as knowing a definite interpretation ahead of time goes. But those of us who think we know the right beliefs for some of the above (e.g., the question on religious views) can go ahead and interpret, and I’ll post the aggregate data on the web so that those with different interpretations of the right answers can interpret differently.
Also, if we find that an unexpected answer to one of the questions on section E correlates with the “best-guess-right” answers to othe other section E questions, and to correct answers on section D, and to e.g. trying to seek information… that’ll be evidence that that unexpected answer might be correct after all. And so, if we eventually develop an “actually yielding rationality scores” version of this test, we could either skip that question, or score that question in the unexpected direction.
It seems like there are two goals here: using our opinions of what beliefs are true to find out whether people are rational, and using our opinions (informed by the test) of whether people are rational to find out what beliefs are true. (We can use some of the information in one direction and some of the information in the other direction, but we can’t use any of the information in both directions, so to speak.) From the latter perspective I think it might be very useful to just ask people a lot of probabilistic questions about “big issues”, and don’t ask them about personal stuff so they can attach the estimates to their screennames. Maybe this needs a way to avoid commitment pressures.
I agree that it might be interesting to know what correlates with getting H&B questions right, but I’m not sure getting H&B questions right translates that well to rationality in general, especially on the right end of the curve.
Re: “Attempts to seek information”
The implication being that it is “irrational” to be lazy.
Something to discuss on those “definiton of rationality” threads, perhaps.
No… the implication is that attempts to seek information might (or might not) correlate with accurate answers in section D, with real-world indicators of “winning”, and with other variables of interest. And that it’d be interesting to know if they correlate.
This is a project that I think really would profit from recruitment of a few psychologists with experience on creating personality test, IQ test or similarly. It sounds a bit like we’re trying to create a new subfield here. Not that I want to sound discouraging, I think it is very important to get the ball rolling and even small, preliminary results could prove to be very useful, but there is probably enough material here to base quite a few academic careers on.
I’ll have to agree with Kaj that a short survey is better for most purposes, but throwing out a long list of ideas first to later hone down to a more efficient one is a good idea.
For example, we could use the CRT, which is very short; or the California Critical Thinking Skills Test (CCTST)
Related to E. 9, one possibility (which I’m not sure is in the standard heuristics and biases tests) might be to exploit the reasonable disagreement type results in asking for people’s opinions on things on which there is significant expert consensus. In such circumstances, we may also want to allow those with greater expertise more leeway to differ from the consensus (where those with no particular expertise should probably just align with it) so maybe a question on expert knowledge of the area would be useful (as objective as possible to avoid self-inflation: majors, advanced degrees etc.).
As steven0461 suggests, if you could could come up with more of them, from different ends of the political spectrum that would be ideal.
Another possibility here might be to identify sets of beliefs that we wouldn’t expect to be correlated except by arbitrary factors like group identification. (I forget the usual example here: would abortion/global warming work?) The tendency for such beliefs to correlate might then serve as a (partial) rationality measure.
Won’t this create a bias toward finding a rationality/centrism cluster?
Are you referring to the final suggestion? If so, I think it should possible to analyze the data to avoid this. There are (very roughly) two ways to get low covariance between opinions: one is to have low variance in opinions generally (i.e. to have middling opinions on everything) , and seems to correspond to centrism. The other is to have a low correlation between opinions (e.g. you could have more extreme opinions, but from both ends of the political spectrum). If you focus on covariance, you’ll capture both, but because correlation is normalized for variance, focusing on that should help avoid “centrism bias” (I think).
Step One should indicate stating hypotheses before checking the data. Great start!
Should C.4 be whether you have caused/been responsible for a car accident? I’ve been crashed into, and have been the passenger in a car responsible for an accident. I’m not sure either of those greatly reflect anything about me.
But they might, statistically. Really good drivers avoid accidents that wouldn’t have been their fault. Really cautious people don’t get into the car with folks that shouldn’t be driving. Really good judges of character pick really good drivers to drive with.
Your skill at not causing accidents, your skill at avoiding accidents that someone else would cause, your skill at evaluating these skills in others, and the number of times you have been in situations that test these skills seems like a lot of information to condense into 1 binary value.
It is—that’s what makes things like ‘rationality’ good heuristics—they bind together a bunch of different things that would be unwieldy to get at separately.
As for the ‘binary’ part, it could as easily be ‘how many accidents have you been in?’ (followed, perhaps, by ‘how many have been determined to be your fault’)
Good heuristics should synthesize relevant information and filter out noise. Counting indicators that should have different weights with the same weight is noise. Causing an accident is worse than failing to avoid one, even though failing to avoid one is bad.
Excellent, now we are using the criticism to improve the question. I would further change to “how many accidents have you caused (or been in) in the last year?”. Or some other set time. And ask how many miles the subject drives per year in the same time period, to normalize. Other demographic information might help here.
SATs are a good test of ‘g’ if you believe in ‘g’ but it might be good to ask for Ravens or another culture fair test from those who have such results in case SATs, largely a vocab and basic math test, reflect dispositions to learning that correlate better with rationality or nerdiness or something than does culture fair IQ.
Good idea—I heartily support this effort. That being said...
I would recommend not making the survey too long, as that reduces the chance that people will actually bother completing it. The longer it is, the likelier it is that only the people who actually have an emotional investment in the subject will finish it, biasing the results.
Once we gather more data, this can be partially avoided by making it into one of those online tests. “How rational are you? Fill this survey and find out.”
Some of the questions on “real-world success” are pretty odd—what does having a driver’s license have to do with success? (Living in an urban environment with good public transport, I don’t have a license simply because almost anything has felt like a better investment of time and money.) Number of best friends probably is more influenced by your personality and relationship style than rationality. If you ask for GPA, you need to provide some sort of a conversion chart between the systems used in different countries. Note also that not all countries place as high an emphasis on GPA as the US does—many Finnish high school and college students have probably never even had any reason to bother calculating their average, so the motivational factor may be lessened. To help correct for this, ask for one’s country in the geographical questions.
For myself, I have a tendency to start reading non-fiction books and then put them down before I’ve finished them but after I feel I’ve already gotten their message. Might be good to ask for the amount of pages read instead of books finished, to adjust for this. (I have read hardly if any non-fiction books to the end in the last month, even though I do think I’ve read several hundred pages worth.) May also ask how many academic, peer-reviewed articles one has read in the last month.
I’m not sure how “how good-looking are you” is useful with fulfilling the stated goals of section E.
If good looks aren’t strongly correlated with the answers to other questions, we can use the responses of groups of test-takers to “how good-looking are you?” to see whether that group, in aggregate, tends to be positively self-deceived. It isn’t definitive, but if e.g. the group of respondants whose questionnaires are low in a particular factor also have the property that 50% of them regard themselves as being in the best-looking 10% of the population, while for the group whose questionnaires are high in that factor, only 10% regard themselves as being in the best-looking 10%… that would be evidence (though not definitive evidence) that the factor correlates with accurate self-assessment.
Not sure whether this is feasible, but could you use the results of this sort of overoptimism calibration to adjust other subjective performance measures for bias? Maybe if you had more of these sorts of question in different domains, and overoptimism were strongly correlated across them all?
Good suggestions re: the questions about reading. Thanks. I’ll change them.
Your comments about real-world successes are good too. Will GPA be more or less country-neutral if I ask for respondants’ percentiles? I’ll throw in an urban vs. rural question next to the question about drivers’ licenses. I agree that “number of best friends” is far from definitive, but then again so is income (not everyone prioritizes money): the idea is to ask about success in hitting many different indicators that some portion of respondants will have aimed for, so as to accumulate many weak indicators (which may together make a stronger indicator) of success in hitting one’s goals.
Do you have other ideas for other questions to include here?
Another nit about drivers’ licenses (full disclosure: I don’t have one, and I live in the USA): from what I’ve seen, drivers’ license as an indicator of “real world success” is a very American phenomenon. Anecdotally, the Europeans I’ve encountered seem significantly less likely than Americans of the same age to have licenses on average, nor is there a stigma (or as much of a stigma) associated with not having one.
Well, not all countries use percentiles, either. Finnish college grades are a number on a scale from 1 to 5 (and high school grades are a scale from 4 to 10), and it depends somewhat on the course and subject how those grades are produced. Some courses, for instance, will pass you if you get 33% of the exam right, others require 50%. In either case, only the grade is recorded, not the percentile.
Rural vs. urban is probably a pretty good control for the license question.
No ideas that I could think from the top of my head, but I’ll comment if anything occurs to me.
You could ask specifically about how important different goals are to people, although this admittedly opens up the possibility of people shifting either their stated goals in response to their stated performance, or vice versa. Having relatively “objective” performance criteria would mitigate the latter, while having the importance questions well before the performance ones could mitigate the latter.
As an alternative it might be possible to combine the two into a single question, something like “How successful have you been in achieving the things that are important to you?” Maybe that’s too subjective, but perhaps it could be made more specific… E.g. “Think about an important goal that you have pursued in the recent past. How successful were you in attaining that goal.” might be workable if the answer options were also specific enough?
Good suggestions re: the questions about reading. Thanks. I’ll change them.
Your comments about real-world successes are good too. Will GPA be more or less country-neutral if I ask for respondants’ percentiles? I’ll throw in an urban vs. rural question next to the question about drivers’ licenses. I agree that “number of best friends” is far from definitive, but then again so is income (not everyone prioritizes money): the idea is to ask about success in hitting many different indicators that some portion of respondants will have aimed for, so as to accumulate many weak indicators (which may together make a stronger indicator) of success in hitting one’s goals.
Do you have other ideas for other questions to include here?
I second the difficulty of having a universal educational achievement measure. In France we don’t put such a great accent on the GPA (I think I had one, but don’t know what it is), and I wouldn’t know how to calculate the SAT equivalent.
(still, it would be a good idea overall)
It’s nice to see that some players cooperate
Ah, yes measuring rationality—a new field is opening up. Job applicants will soon be required.....
Perhaps there is a little sub-group who will refuse to take such a test on the grounds of bias, idiocy and metarationality.
Those are the ones I will employ.