An Anchoring Experiment
The experiment is closed, for the results look here.
In recent discussion I have expressed an opinion that anchoring may, for some quantitative questions, cause the answer to lie further away from the correct value than the anchor itself. For concreteness, let’s suppose that the correct value of a quantity Q is x, and the subject is asked whether Q is greater or lower than y, y > x. My hypothesis is that the anchor moves the subject’s probability distribution up as a whole, including the part which already has been lying above y. Therefore the subjects will positively answer the question “Is Q > y ?” more often than their guess would exceed y if they were just asked to estimate the value of Q with no anchor given. One commenter apparently disagreed. I thought it may be interesting to resolve the disagreement experimentally. (More generally, I would like to see how well LW audience fights the standard biases, and if this experiment turns out successful—which means the number of respondents be greater than, say, five—I would think about posting more of this kind.)
How to participate:
The experiment has two parts.
First, toss a coin to decide whether you belong to the biased group I or the control group II for the first question. If you belong to the group I, look at a comment linked below, which will give you a question of form “is Q is greater or lower than y”, where y is either significantly lower or significantly greater than the correct value of Q. The comment has a form of a typical LW poll. If you belong to the group II, look at different linked comment which asks “what is the value of Q”, and then give your estimate in a subcomment there.
The second part is completely analogical to the first one, only with a different question. If you have participated in the first part within the group I, take part in the group II for the second part, and vice versa. Try to eliminate the irrelevant biases: switch on the anti-kibitzer before looking on the group I questions to avoid being influenced by the votes of others. Don’t read the subcomments of the group II questions before writing down your own.
The hypothesis is that the percentage of the group I respondents answering incorrectly will be greater than the percentage of the group II respondents estimating on the incorrect side of the anchor.
First part: Question for the group I. Question for the group II.
Second part: Question for the group I. Question for the group II.
Uh, some people’s answers are short enough that their numbers are showing up in the ‘Recent Comments’ section. Please make a buffer in your comments prior to the answer if it’s short (I don’t think spaces work, but I’m not sure.) so this doesn’t happen.
Thanks. I have been already given a notice about that and have included a similar instruction in the relevant comments.
Your sample buffer is one character too short.
Fixed.
We should vote down the group II questions as well.
I suggest upvoting the main post and downvoting the group II question you answered.
Edit: At least as a stopgap until prase makes a proper karma balance comment.
My coinflip put me in group II, but since I was reading /comments/ I saw one of the anchors intended for group I, so I decided that for me to continue would bias the results.
In addition I was primed (anchored? terminology?) by the answer of another subject in group II, and this time the problem could not be blamed on my having been reading /comments/.
Nice try, but I doubt the method of collecting responses you have chosen captures the aspects of reality you wish to investigate. Maybe you could redo the collection of responses by having respondents email their responses. (ADDED: rot13ing the response like muflax did works too.) If you do redo the collection of responses, I humbly suggest that you hide the instructions given to group I from group II—e.g. by putting them behind a hyperlink to another page.
Thanks for feedback. I know that it is difficult to avoid seeing the comments of others. The problem with e-mail is that it is marginally less convenient to do, and I was afraid that people would have not enough motivation to respond.
As for hiding the instructions, I don’t see a strong reason. I suppose priming is a strong enough effect to persist even if people know they are being tested for it.
If the coin flip puts me in group II, does not my inadvertently seeing the question intended for group I (e.g., “Is the population of the Central African Republic lower or greater than 20 million?”) de facto reassign me to group I? In other words, after I had been exposed to the anchor (20 million) would not my continued participation in the control group have undermined the purpose of the control group?
ADDED.
I tend to agree, but that’s not it.
Agreed. It would have been better to always answer the group-II question first, whichever one you were assigned to. (That is, switch the part 1 group II question with the part 2 group I question.)
I have misunderstood you and thought you suggest that I hide parts of the original post.
I have expected that people follow on the links in the bottom of the post. If you click on the group II question link, you don’t see the group I question. Of course, if you scroll down, you will see it, but therefore I have warned against reading the comments.
As a positive data point, the instructions worked for me, and I think this is an interesting and well thought-out experiment—the method isn’t perfectly watertight, but that has to be balanced against making it easy and attractive for people to participate, as well as against the cost (in your time and energy) of setting it up. I think you struck a good balance.
Err, I think I screwed this up. I thought if you were in group I for the first part, you’d also be in group I for the second part. I still think this would be more intuitive (just reverse the questions in part 2).
Perhaps, but I am not going to change it now when several answers have been already collected.
Sure, I wasn’t suggesting you should. Given the current situation, I’d change
to
I probably would have tried to use bulleted lists and/or bold to make the text more scannable overall. But maybe my error was atypical and all this isn’t necessary.
From a quick eyeballing of the results so far it does seem that the anchors are having the expected effect, even though we know that we are being anchored.
Anti-kibitzer is acting odd in my Google Chrome browser—it’s not bolding links and all the comments are shown in full, rather than truncated to a single line.
I think you should have posted just the links to the questions and the instructions on which link to follow without explaining what the experiment is about.
That would be certainly a reasonable way to do it, however I suppose that the LW audience is well informed about the biases and able to guess what it is all about even without being explicitly told. Meanwhile I wanted to provide some explanation why I am doing this, and to publish the tested hypothesis in advance (as a measure of honesty).
The effect seems robust enough to survive when people know that they are being tested for it.
Part 1, group II question:
What is the population of the Central African Republic?
Give an estimate in a subcomment. Please begin your answer with “I suppose the correct value is probably” or some other preface of comparable length; if you write just the number, it appears in the Recent Comments bar and can bias other respondents.
I note that I don’t know what the Central African Republic is, but I’m guessing that it’s somewhere in Africa. I’m probably best off estimating that it has the average population of a state in Africa, but I don’t know what this is either. It will be much smaller than the US population (P>.999), but that doesn’t help me....much. I note that this fact in itself implies that I can form a probability distribution. My estimated probability of the Central African Republic being smaller than the US is .999, so I have .999 to distribute within the range 0>450,000,000 (4.5x10^8 is my estimate of the US population).
I don’t think that I should use a hypothesis of complete ignorance here because I think that I would have a greater chance of hearing of it the more people it had, but this isn’t very dependable considering that I don’t follow politics that much. I also note that you may have made the name up, giving it a population of zero. But wouldn’t that render the experiment invalid? I can’t immediately see why. You’re also posting this on April Fools, of all days, and we haven’t had a joke yet, so I weight this option more than the others.
Yet we’re trying to minimize error here, not pick the integer which we believe has the highest probability of being a correct answer. To take this possibility into account, I will multiply my guess of an average nation’s population by .4.
What data can I use to make that guess. Well, I was at a Model United Nations conference once, and I thought that there were about 94 nations present. I suppose that there are about 40 nations not present in the UN, so let’s run 6.5 billion/136. We get 46 million. I will multiply that number by .4 to get 18.7 million.
Reposted: put this in the wrong place the first time.
Upvoted for being the sort of analysis one should do to avoid anchoring. Ironically it ended up much closer to the anchor than to the correct value.
rot13ed: svsgl zvyyvba crbcyr
I suppose the correct value is probably 16 million.
I suppose the correct value is probably around 2 million.
I suppose the correct value is probably 10 million
I suppose the correct value is probably 10 million.
blah blah blah I also would have guessed 5 million.
Probably: around 10 million.
I had no idea there was even a country with that name, 20 million.
I suppose the correct value is probably 10 million. (Incidentally, I have no clue whatsoever, this is a guess)
Zl thrff jbhyq or svir zvyyvba
Edit: scratch that. I was influenced by the first answer I saw on the page… my answer should not count. Leaving it for posterity, below—I’d make it strikethrough if I knew how.
I suppose the correct value is probably 50 million.
I don’t know what the Central African Republic is, but I would guess about 1 million.
I remember looking up the population of Libya recently, which is around 6m, and I feel like the population of Central African Republic is a bit lower. I suppose the correct value is around 3m.
I suppose the correct value is around 5 million. I posted this without looking up any data that might have informed my answer.
I have very little idea but it’s probably around five million?
I note that I don’t know what the Central African Republic is, but I’m guessing that it’s somewhere in Africa. I’m probably best off estimating that it has the average population of a state in Africa, but I don’t know what this is either. It will be much smaller than the US population (P>.999), but that doesn’t help me....much. I note that this fact in itself implies that I can form a probability distribution. My estimated probability of the Central African Republic being smaller than the US is .999, so I have .999 to distribute within the range 0>450,000,000 (4.5x10^8 is my estimate of the US population).
I don’t think that I should use a hypothesis of complete ignorance here because I think that I would have a greater chance of hearing of it the more people it had, but this isn’t very dependable considering that I don’t follow politics that much. I also note that you may have made the name up, giving it a population of zero. But wouldn’t that render the experiment invalid? I can’t immediately see why. You’re also posting this on April Fools, of all days, and we haven’t had a joke yet, so I weight this option more than the others.
Yet we’re trying to minimize error here, not pick the integer which we believe has the highest probability of being a correct answer. To take this possibility into account, I will multiply my guess of an average nation’s population by .4.
What data can I use to make that guess. Well, I was at a Model United Nations conference once, and I thought that there were about 94 nations present. I suppose that there are about 40 nations not present in the UN, so let’s run 6.5 billion/136. We get 46 million. I will multiply that number by .4 to get 18.7 million.
About 25 million people.
5 million
30 million people
Part 2, group II question:
What is the altitude of the highest point in Sweden?
Give an estimate in a subcomment. Please begin your answer with “I suppose the correct value is probably” or some other preface of comparable length; if you write just the number, it appears in the Recent Comments bar and can bias other respondents.
I suppose the correct value is probably 5000 km.
I had no idea what a reasonable height for a mountain might be, but I did half-remember a quote from the Spanish dub of Jurassic Park III about K2 being between x000 km and 9000 km. I knew that K2 was the 2nd highest mountain on the planet, so I thought Sweden’s own highest mountain might be quite a bit smaller and therefore wrote down 5000 km as my guess. Except that, lo and behold, it turns out that I misremembered meters as kilometers! Great, now I’m gonna have the most inaccurate answer in this subthread… so much for my attempt to at least be in the correct order of magnitude.
5000 kilometres ?
Yes, 5000000 meters. Like I said, I didn’t realize I had screwed up until I looked at the other posts, which was after I had written my answer down.
I don’t know anything about Sweden, nor about geology, so I will not be surprised to find that the following is very wrong. I guess that the highest point in Sweden is about 1800m above sea level.
I’d guess around 8,000 feet (I seem to recall Sweden having only smaller mountains)
Oh darn that question, I climbed it last summer.
Rot13: Fyvtugyl zber guna gjb gubhfnaq naq bar uhaqerq zrgref nobir gur frn. Gur zbhagvna vf fuevaxvat.
I suppose the correct value is probably about 10,000 feet. I base it on not knowing much about Sweden, but seeing bumps on maps, and guessing that they don’t have anything any bigger than 2⁄3 of Mt. McKinley, which probably served as an anchor for me, so I’m not sure if I actually belong in group II any more.
Edit: The number I guessed was actually closer to half a McKinley. So much for memory.
I suppose the correct value is probably around 3000 m.
I know of no large mountains to be found in Sweden, so I’m guessing what seems to be a reasonably low number.
My guess of the answer to this question, bearing in mind that I haven’t studied geography since elementary school and am very unsure at to the order of magnitude, is 3600 feet.
Some notes on my reasoning:
I was originally going to guess an answer of 10000 feet on the basis that I vaguely remembered a Nickleback song indicating that commercial airplanes go up even higher, but thought without looking that the lyrics might have been “join the mile high club at 20000 feet” and so disregarded this answer as biased (but apparently the lyrics to “Rock Star” are “37000 feet” instead, so this was a mistake). I then figured that 6x the height of the Sears Tower would be a somewhat reasonable height for a mountain, and thought that the Tower was either 300ft or 600ft tall, eventually deciding on 600ft (actual height: 1451ft). 6(600) = 3600.
Absolutely no idea what the answer is, so I’ll guess 2000 feet.
I have no concrete preexisting knowledge about the scale of altitudes, or the geography of Sweden, including whether or not it contains mountains. I am therefore somewhat arbitrarily choosing one-half of the value which Wikipedia labels “high altitude”, namely 1200 meters. I also note that on first reading I missed “the highest point” and worked on answering the typical occupied altitude of Sweden, and had some confusion with Switzerland.
I suppose the correct value is probably...
I’m trying to remember anything about the geography of Sweden and i’m drawing a blank. Hmm, i know that the highest mountains in the continental US are about 14,000 feet. I remember the highest mountains in Europe are a few thousand feet higher than that, but i think those were probably in the Alps. At the moment i can’t remember hearing much about famous Scandinavian mountain ranges, but then again they could be higher than anything in the continental US and still not be as famous as the Alps, right? For the sake of having an answer i’ll guess about 14,000ft.
I suppose the correct value is probably..................here are many characters for padding............… 100 meters above sea level.
Well, I’m constantly disappointed by how low the highest peaks are here in the UK, and tend to imagine almost all other nations as having much higher ones. I also know that Norway is extremely mountainous, and that Sweden is much less so, but might still be mountainous around the edges or up north (they hosted the winter olympics one year, after all.) So I’m going to guess 2000m. This is consistent with having been “anchored” by the anchor, but honestly, I think my own weird logic of taking the UK’s highest mountain and “adding a bit on” is to blame.
No idea, but my impression of Sweden isn’t very mountainous so I’ll guess 3000m (~10000 feet)
I would guess somewhere around 12,000 feet.
Thinking about the question in the context of my current education on geography (a disturbingly large portion of which is from the 1632 series of books), I would imagine that Sweden includes some reasonably tall mountains—but not record-breakingly tall. As ignorant as I am, I’m going to use 5000 feet as a benchmark for an ordinary mountain, and guess 12,000 feet.
No clue, but anchoring off my state and adjusting downward on the belief that it’s a high-than-average state: 10,000 feet
Knowing virtually nothing about the geography of Sweden, I’ll guess 10,000 feet.
Probably in the low four digit meter range, I guess 1200m.
I looked at a map some years ago, but I don’t really remember it. I think it’s a bit more mountainous than Ireland or Scotland, so I’m going to guess 1500 m.
i suppose the correct value is probably 10,000 feet.
If I remember right, the Scandinavian peninsula is rugged but not particularly mountainous in absolute terms, and the Swedish part is less so than the Norwegian. Yrg’f fnl friragrra uhaqerq naq svsgl zrgref.
Edit: Rot13′d.
I seem to recall that Sweden is “somewhat mountainous”, so I suppose that the correct value is around 2000m.
I accidentially looked at the first comment, so my guess could be already biased, however, I think I remember that I hat a rough figure already in my head when I saw the other number, so probably the guess is still valid. I think 1500 m
I suppose the correct value is about 5 km.
I guess that the highest point in Sweden is probably about 1000 feet above sea level.
Let’s see, how would I go about guessing? I forget if Sweden is supposed to be mountainous, and looking up stuff is probably outside the intent of the experiment, so… I’ll give a range. Probably between 500ft and 10,000ft. If you want, for the purposes of the experiment, a single number, let’s say 5000ft.
I suppose the correct value is … 1800m
3500feet.
I guess the following number:
5000 feet
1000 ft
900 meters.
7000 ft
(about 7⁄3 km)
500 meters
Part 2, group I question:
Is the altitude of the highest point in Sweden lower or greater than 3500 metres (11500 feet)?
Vote by upvoting the subcomments. This comment serves as karma balance for the votes. Downvoting it not only spares the need to have a separate karma balance, but also prevents people from seeing the anchor by accident.
Upvote if your answer is lower.
Upvote if your answer is greater.
I would suppose that the answer is greater.
I first think Sweden is near the Alps, and then remember it being near the Northish area of Europe from a map...maybe. When I think of Sweden, I think of outdoorsey things, so I think of mountains. Bugger: the anchor’s making me feel like the problem shouldn’t get as much attention as the first one because I can just pick lower or greater.
I note that you’re probably not going to have an elevation greater than 11500 ft unless you have more jagged mountains (great technical term, there :D) rather than smooth ones—Rockies, not Appalachians. I’m going to guess greater, since I think that a certain character from a book was from Sweden and found mountains familiar (and my image of the scene made them feel like jagged mountains.
I didn’t put as much effort into this one as I did into the first one. It’s probably due to a mixture of hunger, burnout, and the fact that I have other things that I want to do.
I would suppose that it is greater
If you have not done it already, please upvote this comment and downvote this comment (to ballance the karma), so that the number of upvotes for each variant is equal to the karma of the corresponding comment. (This is the standard way how polls are conducted on LessWrong.)
rot13ed: vg vf ybjre
Part 1, group I question:
Is the population of the Central African Republic lower or greater than 20 million?
Vote by upvoting the subcomments. This comment serves as karma balance for the votes. Downvoting it not only spares the need to have a separate karma balance, but also prevents people from seeing the anchor by accident.
Upvote if your answer is lower.
Upvote if your answer is greater.