This post summarises the results of the experiment which tested how anchoring works on the LW audience. Here is the original post which describes the experiment in more detail. The experiment was supposed to decide between two ways of how anchoring may work. The first hypothesis is that the subject always starts from the anchor and continues in the direction of his/her unbiased estimate, but doesn’t go far enough. The alternative hypothesis is that anchoring shifts the centre of the subject’s probability distribution towards the anchor, and the whole distribution moves along.
To illustrate the difference, consider the first experimental question, which was about the population of the Central African Republic. The correct value (i.e. the estimate for 2009 listed on Wikipedia) is 4,422,000. The anchor which I have offered here was 20 million. Now, if the first hypothesis is true, the people who, in their unbiased state of mind, would guess less than 20 million, would slide down starting from the 20 million value and stop prematurely; their guesses will be attracted towards the anchor, but not across. The distribution of the biased guesses would be narrower and overall closer to 20 million than the unbiased distribution, but the probability of answering whatever number lower than 20 million would not be changed by anchoring. On the other hand, if the second hypothesis holds, the biased group should guess more than 20 million more often than the control group.
The actual results are such:
Group I (biased; 36 answers collected)
more than 20 million: 15 (41.7%)
less than 20 million: 21 (58.3%)
Group II (control; 16 answers collected)
more than 20 million: 3 (18.7%)
less than 20 million: 13 (81.3%)
20 million: 1 (6.3%)
The second question asked for the altitude of the highest point in Sweden (2140 m / 6903 ft). The anchor was 3500 m or 11500 ft (there is about 5 m / 18 ft difference between the values, but I wanted both the metric and the imperial anchor to be round numbers). Here the results are:
Group I (biased; 24 answers collected)
more than 3500 m: 9 (37.5%)
less than 3500 m: 15 (62.5%)
Group II (control; 30 answers collected)
more than 3500 m: 5 (16.7%)
less than 3500 m: 13 (83.3%)
The results seem to favour the second hypothesis.
Some more remarks: The participants were expected to change the groups between both parts and the numbers should reflect that, however, six and eight answers are missing from the group II summaries. Few people (about 16% and 33% actually) thus refused to guess the concrete number although they voted in “greater/lower than anchor” questions. It may skew the results, although I don’t see in which direction.
There were few weird answers, too. The altitude of the Sweden’s highest summit was reported to be both 100 m and 5000 km. Those can be simply interpreted as statistical deviations from common sense* (or typos), however I started to doubt whether all participants were serious. (Which leads to a moral: If you intend to post a survey and be certain about its accuracy, don’t do it on the 1st of April.)
Finally, I would like to thank all the commenters who had pointed out several technical problems with the test (such as the answers appearing in the “recent comments” bar).
*) The 100 m guess may even be reasonable: The summit of Yding Skovhøj, the highest point of extraordinarily flat Denmark, lies only 175 metres above the sea.
An Anchoring Experiment: Results
This post summarises the results of the experiment which tested how anchoring works on the LW audience. Here is the original post which describes the experiment in more detail. The experiment was supposed to decide between two ways of how anchoring may work. The first hypothesis is that the subject always starts from the anchor and continues in the direction of his/her unbiased estimate, but doesn’t go far enough. The alternative hypothesis is that anchoring shifts the centre of the subject’s probability distribution towards the anchor, and the whole distribution moves along.
To illustrate the difference, consider the first experimental question, which was about the population of the Central African Republic. The correct value (i.e. the estimate for 2009 listed on Wikipedia) is 4,422,000. The anchor which I have offered here was 20 million. Now, if the first hypothesis is true, the people who, in their unbiased state of mind, would guess less than 20 million, would slide down starting from the 20 million value and stop prematurely; their guesses will be attracted towards the anchor, but not across. The distribution of the biased guesses would be narrower and overall closer to 20 million than the unbiased distribution, but the probability of answering whatever number lower than 20 million would not be changed by anchoring. On the other hand, if the second hypothesis holds, the biased group should guess more than 20 million more often than the control group.
The actual results are such:
Group I (biased; 36 answers collected)
more than 20 million: 15 (41.7%)
less than 20 million: 21 (58.3%)
Group II (control; 16 answers collected)
more than 20 million: 3 (18.7%)
less than 20 million: 13 (81.3%)
20 million: 1 (6.3%)
The second question asked for the altitude of the highest point in Sweden (2140 m / 6903 ft). The anchor was 3500 m or 11500 ft (there is about 5 m / 18 ft difference between the values, but I wanted both the metric and the imperial anchor to be round numbers). Here the results are:
Group I (biased; 24 answers collected)
more than 3500 m: 9 (37.5%)
less than 3500 m: 15 (62.5%)
Group II (control; 30 answers collected)
more than 3500 m: 5 (16.7%)
less than 3500 m: 13 (83.3%)
The results seem to favour the second hypothesis.
Some more remarks: The participants were expected to change the groups between both parts and the numbers should reflect that, however, six and eight answers are missing from the group II summaries. Few people (about 16% and 33% actually) thus refused to guess the concrete number although they voted in “greater/lower than anchor” questions. It may skew the results, although I don’t see in which direction.
There were few weird answers, too. The altitude of the Sweden’s highest summit was reported to be both 100 m and 5000 km. Those can be simply interpreted as statistical deviations from common sense* (or typos), however I started to doubt whether all participants were serious. (Which leads to a moral: If you intend to post a survey and be certain about its accuracy, don’t do it on the 1st of April.)
Finally, I would like to thank all the commenters who had pointed out several technical problems with the test (such as the answers appearing in the “recent comments” bar).
*) The 100 m guess may even be reasonable: The summit of Yding Skovhøj, the highest point of extraordinarily flat Denmark, lies only 175 metres above the sea.