Political Debiasing and the Political Bias Test
Cross-posted from the EA forum. I asked for questions for this test here on LW about a year ago. Thanks to those who contributed.
Rationally, your political values shouldn’t affect your factual beliefs. Nevertheless, that often happens. Many factual issues are politically controversial—typically because the true answer makes a certain political course of action more plausible—and on those issues, many partisans tend to disregard politically uncomfortable evidence.
This sort of political bias has been demonstrated in a large number of psychological studies. For instance, Yale professor Dan Kahan and his collaborators showed in a fascinating experiment that on politically controversial questions, people are quite likely to commit mathematical mistakes that help them retain their beliefs, but much less likely to commit mistakes that would force them to give up those belies. Examples like this abound in the literature.
Political bias is likely to be a major cause of misguided policies in democracies (even the main one according to economist Bryan Caplan). If they don’t have any special reason not to, people without special knowledge defer to the scientific consensus on technical issues. Thus, they do not interfere with the experts, who normally get things right. On politically controversial issues, however, they often let their political bias win over science and evidence, which means they’ll end up with false beliefs. And, in a democracy voters having systematically false beliefs obviously more often than not translates into misguided policy.
Can we reduce this kind of political bias? I’m fairly hopeful. One reason for optimism is that debiasing generally seems to be possible to at least some extent. This optimism of mine was strengthened by participating in a CFAR workshop last year. Political bias seems not to be fundamentally different from other kinds of biases and should thus be reducible too. But obviously one could argue against this view of mine. I’m happy to discuss this issue further.
Another reason for optimism is that it seems that the level of political bias is actually lower today than it was historically. People are better at judging politically controversial issues in a detached, scientific way today than they were in, say, the 14th century. This shows that progress is possible. There seems to be no reason to believe it couldn’t continue.
A third reason for optimism is that there seems to be a strong norm against political bias. Few people are consciously and intentionally politically biased. Instead most people seem to believe themselves to be politically rational, and hold that as a very important value (or so I believe). They fail to see their own biases due to the bias blind spot (which disables us from seeing our own biases).
Thus if you could somehow make it salient to people that they are biased, they would actually want to change. And if others saw how biased they are, the incentives to debias would be even stronger.
There are many ways in which you could make political bias salient. For instance, you could meticulously go through political debaters’ arguments and point out fallacies, like I have done on my blog. I will post more about that later. Here I want to focus on another method, however, namely a political bias test which I have constructed with ClearerThinking, run by EA-member Spencer Greenberg. Since learning how the test works might make you answer a bit differently, I will not explain how the test works here, but instead refer either to the explanatory sections of the test, or to Jess Whittlestone’s (also an EA member) Vox.com-article.
Our hope is of course that people taking the test might start thinking more both about their own biases, and about the problem of political bias in general. We want this important topic to be discussed more. Our test is produced for the American market, but hopefully, it could work as a generic template for bias tests in other countries (akin to the Political Compass or Voting Advice Applications).
Here is a guide for making new bias tests (where the main criticisms of our test are also discussed). Also, we hope that the test could inspire academic psychologists and political scientists to construct full-blown scientific political bias tests.
This does not mean, however, that we think that such bias tests in themselves will get rid of the problem of political bias. We need to attack the problem of political bias from many other angles as well.
The problem here is who will have the authority to decide what viewpoint is to be considered unbiased.
Let’s assume a completely unbiased viewpoint about a very controversial political topic actually existed. Wouldn’t most people see it as biased in favor of their opponent’s side?
Cool! I’ve been desperate to see a rationality test and so make improvements in rationality measurable (I think the Less Wrong movement really really needs this) so it’s fantastic to see people working on this. I haven’t checked the methodology yet but the basic principle of measuring bias seems sound.
So, your biases are similar to Stefan’s, congratulations.
No, it’s total accuracy on factual questions, not the bias part...
More importantly, don’t be a jerk for no reason.
See my comment here on just how factual the “facutal” questions are.
I’m generally a proponent of nuclear power but you might want to make your
“Can radioactive wastes from nuclear power be safely disposed of?”
question clearer as my first thought was “define ‘safely’”.
I’m normally the one saying that sealing it in glass, wrapping it in steel, concrete, steel and then burying it in a mountain above the water table in the middle of nowhere constitutes safe for reasonable purposes but it’s also pretty fair for critics to say that we have so little experience of really long term construction projects or what future conditions will be like (perhaps a settlement will end up there thousands of years down the line for no good reason) that just pointing to a review paper doesn’t really cut it.
If you’d said “reasonably safe” or “safe enough that it’s unlikely to cause problems unless someone intentionally mines into it a thousand years from now and starts selling glowing jewlery to all the locals” it would have been a clear yes but I couldn’t answer that without disclaimers.
I just looked at this test and had a problem with the very first question. It asks if you are strongly conservative or strongly liberal. What? I’d expect a high percentage of libertarians taking the test.
Other problems: Policies that are pro-immigration: If I believe in some policies that are pro-immigration and some that are not (like if I believe that certain groups of immigrants are bad and others aren’t), what do I answer? Am I supposed to answer the question as if it means “are you in favor of unrestricted immigration?” (which it often means in political discourse) or actually answer what it literally says?
Actually, that’s a general problem with the whole survey. There are questions which might have one meaning literally, but which mean something other than their literal words in political discourse. The global warming question doesn’t say ”… to such a degree that in order to fix it, we should put drastic policies into place”, but that’s what the question usually means. You’re going to end up misdetecting bias when you’ve actually detected ability to read subtext and context.
What am I supposed to answer for questions about subjects on which I have no opinion? There isn’t a “no opinion” answer on the question about whether I support tough on crime policies. And I have no idea how many percentage points renewable energy grew by.
There’s also a problem whcih I’ve seen in lots of other surveys. The survey question consists of a statement followed by the actual question, phrased in a way which indicates that you’re supposed to assume the statement is accurate. What do you do if you don’t consider the statement to be accurate? (“Advocates of American capitalism argue that global technological and economic progress is largely due to American ingenuity.” Literally true since it fails to state how many such advocates, but I don’t believe what it is probably intended to say. “Taxes are much lower in the US than in the average EU country. If lower taxes meant more wealth, you would therefore expect the US to be richer, in the sense of having a higher GDP per capita, than most EU countries.” I would not “therefore” expect any such thing, because I would only expect it if other factors are equal or favorable for the US and the statement hasn’t stipulated that.)
I think at some point you hit a problem with hypocrisy rather than bias. The arguments people make and the beliefs they claim very often don’t align with their actual contingent predictions and preferences.
The basic popular belief “I am not responsible for improving this aspect of the universe” is perhaps best handled not so much as an unconscious bias, but as an unstated motivating belief that drives biases.
Overall fantastic article. One small request:
Citation needed
Thanks!
I read that in a paper by Dan Kahan on bias, but have been unable to find it since. I hope I don’t misremember, but that that was exactly what he said. In any case, I’ll notify you in case I find it.
Can you expand on what evidence there is for this?
The very first question I got (not sure if the order is randomized) was about global warming. Instant priming: Okay, this quiz is going to be about scientific consensus. After that I was trying to guess what scientific consensus said.
That question breaks your quiz.
I’m uncomfortable with the question distinguishing between “richer than most” and “richer than nearly all”. Logically if I answer “richer than most” is true.
There justification of the answer makes little sense as you have to count the number of countries which are richer to find out and not take the EU average.
Yes, I admit some of the questions could have been better phrased. If I do another test, as I hope to, I’ll try to crowdsource this. It would have been easier to come up with good questions if I had had social scientists and scientists in relevant fields on board. Also, I think that would minimize unclarities, and so on (more eyes, etc).
That said, we did a fair amount of pre-testing on Mechanical Turk and on friends.
I had an issue with the phrasing of “are GMOs good for humans”, that’s a different claim than the linked article makes, which is that GMOS are “good for human health”. I put I don’t know because we don’t know what the environmental impact will be in the long run, and if that will be a net positive or negative for humans.
I just took the test. Looking at the official answers, I have to say that the test probably says more about the test makers’ bias then my own. For example, a number of the answers cite “scientific consensus”, which is a rather dubious concept philosophically especially in areas like the global warming and GMOs where there is reasonable suspicion that the “consensus” is politically manufactured, or even worse “economic consensus”, a.k.a., the economists we cherry-picked all agree.
It doesn’t help that some of the economics questions are ambiguous: “Did the Obama administration’s 2009 stimulus package reduce or increase unemployment?” are we including the effects on the economy of borrowing the money to pay for the stimulus?
Another example is the World Giving Index. While the answer, that the US gives more then European states is probably true, the fact that the index has the US tied with Myanmar is extremely strong evidence that the index is BS.
Whether or not the index produces that effect seems to be a fairly objective question. If conversatives get this right but biased liberals get it wrong, this indeed shows bias.
Analogy: I describe a real-life situation of a police officer shooting a suspect. I then ask people what they think the race of the police officer and suspect are. Because I am referring to a specific real-life case, the question has a single, factual, answer, and people’s answer is either correct or not.
Yet I can manipulate the question to show liberal bias or conservative bias, my choice, simply by which case I choose to ask the question about.
The best way to ask that question to legitimately detect bias would be to choose a typical case, and to assume that people who haven’t heard of the specific case would answer depending on the facts about typical cases.
And in this situation, a typical case would be an index of X that accurately measures X. Choosing an index that doesn’t accurately measure X would skew the ability to use that question to detect bias, since I expect that unbiased people who haven’t heard of the index in question would answer based on an accurate measure of X.
That’s an interesting one—I think black people are disproportionately at risk of being killed by the police in the US, but about as many white people as black get killed.
|the fact that the index has the US tied with Myanmar is extremely strong evidence that the index is BS.
Here’s a bit more information; alternatively, you can read the report yourself.
TLDR, the reports are based on self-reporting, the number of people giving is weighted as heavily as the amount given, and giving to religious charities (like Buddhist monks) counts, but yes, Myanmar has a lot of people giving money to other people.
The question isn’t whether the index is BS but what signal the judgement communicates. You don’t learn that by investigating the index in detail but in seeing correlations between the answer to the question and other answers.
(From the test)
Or media bias has a negligible effect, but people randomly encounter more of one side than the other. Do the scores follow a bell curve around no bias?
Hm.. I’ll just ask here for the questions I didn’t get right.
I choose “we don’t know” because I genuinely don’t know, but in all honesty the thing that bothers me is that GMO is a pretty big umbrella. I’m assuming it first means that it’s something grown in a lab and not human selection of which fruits were tastier/bigger/more colourful than others. And in the lab scenario, what was modified? Disease resistance? Vitamin contents? Rat poison in the organism’s blood/juice?
Words are harmful.
What is even “conventional” farming? Giving the plant plenty of sun and water, or spraying Agent Green (Agent Orange’s nature-friendly brother) everywhere? (I believe the proper word is insecticide)
I don’t get this question at all.
Words are harmful THE MOVIE II: harmful.cat-v.org
Waaait a sec.. how can this be true? This seems counterintuitive: the average US citizen would probably have less high-income jobs available to them if not less jobs overall as they have to compete with immigrants too now.
And that assumes all things being equal. In reality, we don’t really know what a “high educated foreign worker” implies, and because things are most likely not going to be equal, I have a feeling foreign workers would work for less money and perhaps even in worse conditions, so this question is to be honest kind of disconnected from reality.
Wouldn’t there be a gradual switch to more covert, less-likely-to-get-caught methods with such a severe punishment? Clearly the punishment has some effect.
Lump-of-labor fallacy. The number of jobs in the economy is not fixed; the more people you have that are good at doing stuff, the more stuff gets done overall.
1) Would a near-infinite number of such workers also be beneficial, or is there any number where adding more foreign workers would not be beneficial?
2) If your answer to 1) is “yes, there is such a number”, how do you determine what it is, and that we haven’t reached it?
Well, I suppose that natural resource constraints do apply eventually...
Never mind resource constraints. Suppose a billion plumbers move to the US. By your and by “economists’” reasoning, this will not drive down the price of hiring a plumber at all, since although competition will reduce the price, the economy is not fixed, and more plumber jobs will be available, compensating for this.
Furthermore, if adding more people doesn’t reduce the number of or the pay for jobs, then it necessarily follows that removing people shouldn’t increase the number of or the pay for jobs. Do you seriously believe that if half the plumbers dropped dead, the price of hiring a plumber wouldn’t go up, and the remaining plumbers wouldn’t find it easier to get jobs?
If it’s unrelated to the plumbing job, one would expect the increase in wages to be transient, as others reskill into plumbing. (Current licensing requirements often require five years of training, but would hopefully be modified in such a situation.)
Talking about a specific trade obscures the general model. A plumber produces a service, but consumes many goods and services. One can imagine America with twice the number of people in it, or America with half the number of people in it (i.e. America in 1950). Prices then and now seem roughly the same in a large number of categories (higher in some, and smaller in others—living in cities that can’t expand anywhere but up is more expensive now than then, but goods and services that benefit from scale are considerably cheaper now than then), particularly when one considers wages. The main wage gains seem to be in industries that have larger returns to scale (things like software).
The primary economic costs with immigration are the switching costs of reallocating people and jobs. (Cultural and social costs deserve a separate discussion.)
Let me try another argument.
Would you rather:
1) Pay taxes to pay for a plumber’s schooling, starting with kindergarten or 2) Have foreigners pay taxes to pay for a plumber’s schooling, and then have the plumber immigrate to your country?
It costs a lot of money to educate a child. When you let in an educated immigrant, your country gets all the benefits of that immigrant’s education without having to pay for it.
The original question compares extra immigrants to doing nothing. It doesn’t compare extra immigrants to extra natives. Your suggestion that extra immigrants are better than extra natives is irrelevant to the question, even if true.
(And if I was a plumber, I wouldn’t want either extra immigrant plumbers or extra native plumbers. There is a reason that cartels try to limit the number of people in a profession.)
I think the two words missing in this discussion are “short-term” and “long-term”.
The question about immigration specifies that immigrants arrive each year, not that there is a one time influx of immigrants that ends. So it’s a continuous stream of short-term effects. The long term equilibrium that happens once society adjusts to the immigrants is never reached because there are always new immigrants to which society hasn’t adjusted yet.
There’s an important difference between effects caused by a surprising change (sudden influx) and the results at equilibrium (ongoing expected rate). The long run is just the sum of the short runs, but the some of those short runs include reactions to previous short runs and expectations of future short runs.
Perhaps “immediate reaction to a change” and “equilibrium” are better terms than “short-term” and “long-term”.
In the equilibrium / long-term, there are very few plumbers who remain solely plumbers throughout their lives. Many will switch roles according to demand and aptitude. So the question isn’t about influx of plumbers, but of humans who can change and adapt.
If you’re continually adding plumbers at a constant rate, you will get an equilibrium. But the equilibrium you get will be different from the one you’ll get if you had a one-time influx of plumbers and the market compensated for that. You’ll get an equilibrium where the influx of plumbers continually drives the prices down and the compensating factors continually drive the price back up. Exactly where this equilibrium falls will depend on the relative rates of each part, and it may, in fact, be a net downwards effect.
Please fix your link.
(Done.)
Isn’t that the job-makers that decide how many jobs are available? It doesn’t really matter how many people are lined up for a particular type of job.
I took the test:
I got the same number of correct answers, and knew while I was taking it that I did not know the answers to many of them. 0% political bias, though.
One problem with a test like this is that you might make sure that you don’t give biased answers while you’re taking the test, but in real life you might use biased answers without thinking about it.
That depends a bit on the enviroment. There are cases where people care very much of signaling group alliance. On the left you find people competing on being more ideological pure than the next person.
I had one experience with an extremly smart person, which politically influential parents and maybe a future political career on her own who once quite explicetly said, that’s what she was doing after she read a room wrongly and excused for that (it wasn’t a even a public event but party internal).
I agree that phrase of mine might a bit too strong. But I think a lot of cynics under-estimate the degree to which people want to be rational and unbiased.
I didn’t get this anecdote, which sounded interesting.
Given that it includes individuals that I don’t want to be identified publically, I put value on anonmysation I will write it up with more detail and send it to you privately.
Great post.
Another great resource is the ABC’s fact checker, for those political news watchers to see whether the politicians they agree with or disagree with are factually correct or incorrect.
And, to actively compare one’s ideological beliefs with theoretical truisms, such as those in game theory
Thanks! Yes, I’m actually working on fact-checking—or rather argument-checking—as well. Here are some posts on that. It’s a related but different theme, both falling under the general concept of political rationality, which I talked about at the LW Community Weekend Berlin and EA Global Oxford.
Or to see if your biases agree with the ABC fact checkers’.
See also.
I have to wonder, does Paul Krugman have no self-awareness, or is this his attempt at subtle self-deprecating humor.
He does have self-awareness, but he already sold his soul...