Extreme updating: The devil is in the missing details
Today Ed Yong has a post on Not Exactly Rocket Science that is about updating—actually, the most extreme case in updating, where a person gets to choose between relying completely on their own judgement, or completely on the judgement of others. He describes 2 experiments by Daniel Gilbert of Harvard in which subjects are given information about experience X, and asked to predict how they would feel (on a linear scale) on experiencing X; they then experience X and rate what they felt on that linear scale.
In both cases, the correlation between post-experience judgements of different subjects is much higher than the correlation between the prediction and the post-experience judgement of each subject. This isn’t surprising—the experiments are designed so that the experience provides much more information than the given pre-experience information does.
What might be surprising is that the subjects believe the opposite: that they can predict their response from information better than from the responses of others.
Whether these experiments are interesting depends on how the subjects were asked the question. If they were asked, before being given information or being told what that information would be, whether they could predict their response to an experience better by making their own judgement based on information, or from the responses of others, then the result is not interesting. The subjects in that case did not know that they would be given only a trivial amount of information relative to those who had the experience.
The result is only interesting if the subjects were given the information first, and then asked whether they could predict their response better from that information than from someone else’s experience. Yong’s post doesn’t say which of these things happened, and doesn’t cite the original article, so I can’t look it up. Does anyone know?
I’ve heard studies like this cited as strong evidence that we should update more; but never heard that critical detail given for any such studies. Are there any studies which actually show what this study purports to show?
EDIT: Robin posted the citation. The original paper does not contain the crucial information. Details in my response to Robin.
EDIT: The original paper DOES contain the crucial info for the first experiment. I missed it the first time. It says:
.. a woman was escorted to the speed-dating room and left to have a 5-min private conversation with the man. Next, the experimenter escorted the woman to another room where she reported how much she had enjoyed the speed date by marking a 100-mm continuous “enjoyment scale” whose end points were marked not at all and very much. This report is hereinafter referred to as her affective report.
Next, a second woman was given one of two kinds of information: simulation information (which consisted of the man’s personal profile and photograph) or surrogation information (which consisted of the affective report provided by the first woman). The second woman was then asked to predict (on the enjoyment scale) how much she would enjoy her speed date with the man. This prediction is hereinafter referred to as her affective forecast.
After making her prediction, the second woman was shown the kind of information (simulation or surrogation) that she had not already received. We did this to ensure that each woman had the same information about theman before the actual speed date. The only difference between the two conditions, then, was whether the second woman had surrogation information or simulation information when she made her forecast.
Next, the second woman was escorted to the dating room, had a speed date, and then reported how much she enjoyed it (on the enjoyment scale). This report is hereinafter referred to as her affective report. The second woman also reported whether she believed that simulation information or surrogation information would have allowed her to make the more accurate prediction about the speed date she had and about a speed date that she might have in the future.
The original paper is here: http://www.sciencemag.org/cgi/content/abstract/323/5921/1617
I just noticed a study that might be relevant here cited in a classic Social psychology book I was reading (Baron, Byrne). The article they refer to is Graziano et al. “Social influence, sex differences, and judgments of beauty : putting the Interpersonal back in interpersonal attraction”, 1993 and the result relevant here is that when women are shown an assessment of a man by some other woman, their own assesment moves towards it. This would make the result discussed here just a corollary of a self-fulfilling prophecy. I do not have time to read either of the articles at the moment but it would probably do some good if someone looked over the Graziano paper and verified whether or not it is relevant?
Thanks much!
The paper says: “Ironically, 75% of the women believed that simulation information would have allowed them to make a more accurate forecast about their date with the man they met, and 84% believed that simulation information would allow them to make a more accurate forecast about a future date with a different man.”
It doesn’t say when women were asked this question. So the crucial information is missing.
The experimenters misunderstood their own experiment. They dutifully reported the manner in which the “data” was gathered, but failed to realize that the reported confidence of the subjects in the two types of information was actually the critical data.
In the second experiment, the experimenters also did the wrong thing: They asked judges which type of information would be more useful to them, descriptions of the 3 personality types, or reports written by people describing their responses. The article does not say that the judges were shown the descriptions of the 3 personality types. Thus, again, we can suppose that the judges decided that knowing the 3 personality types would be more useful was based on an erroneous supposition that the descriptions would contain more information than they actually did.
Also, the experimenters used a separate group of subjects to rate whether experiential or “simulation” information would be more valuable to them, than the group of subjects that they used to determine which type of information was more valuable. Why? They increased the expense of their study by 50% in order to make it less convincing. This makes no sense to me.
So this study is intriguing, but not convincing.
Gilbert’s book, Stumbling On Happiness, describes the research. People’s ability to predict what will make them happy is even WORSE than your post makes it sound. Much worse.
In fact, it really does take an entire book to properly explain just HOW bad we are at guessing what will make us happy, although one could argue that near vs. far thinking has an awful lot to do with it.
(I routinely find that people who have difficulty setting or achieving goals (or who achieve them and aren’t satisfied), are people who haven’t immersively imagined what it would be like to live day-to-day with the getting or having of the goal. Immersive imagination, using “ideal day” exercises (e.g. imagining in present-tense detail what it would be like to live through a day where you have achieved your goal) usually provides surprising feedback on whether the goal is actually a good idea, and what modifications might need to be made.)
I am a big believer in this idea of realistic or immersive imagining. I submit that many people in the LW empirical personspace cluster spend far too much time on unrealistic fantasizing (e.g. through reading sci-fi/fantasy books), and that this is highly detrimental to their well-being. I used to live in a fantasy world myself; I’m slowly trying to break the habit.
When I tell people I think they should cut down on their consumption of fantasy novels they say “No, it’s important to be imaginative!” Absolutely, but it’s way more important to imagine realistic outcomes than to imagine castles in the air (I have a soft spot for the idea of cities on the water).
Immersive imagining is probably a good example of an important rationalist technique. I suspect it’s hard to do well, but skill can be achieved through training.
Immersive imagining sounds to me like Detached Detail. Even if your imagined details are “realistic” and “immersive”, those qualities do not move them outside of the realm of Detached.
Compare your “I like seasteading but not castles in the air.” statement with a claim “I like medieval reenactment but only with historical accuracy.” It seems clear to me that this is merely a preference over fantasies, not a fundamental difference in thought process.
I was very unimpressed by the case made in Gilbert’s book.
Why?
I agree. I don’t understand the praise the book has received. I found the reasoning in the book very sloppy, filled with huge gaps in the logic and more obvious alternate explanations for experimental results that were not even mentioned.
I don’t expect the rigor of a research paper in a popular science book, but even popular science books have standards. I’m sure his papers fill in all the gaps in the book, but if there are multiple obvious explanations for an experimental result and you’re going to tell your readers how to interpret the results, you should at least say why the other obvious (sometimes more obvious) interpretations are less plausible or why the preferred interpretation is so compelling—even in a popular science book.
For some examples of what I mean, see this Amazon.com review.
Actually, the point of my post is that I am unconvinced that the study is meaningful.
Yes, it’s a great book. Read it last month and loaned it to a friend who loved it too. Stupid title though, makes it sound like a self-help book, and the cover isn’t any better.
The trouble with making these predictions really is lack of information. If you were told everything about an experience (IE you experienced it! Maybe virually) then probably your guess would be better than the advice of someone who has. But it’s impossible to do that. Spoken word certainly don’t have the bandwidth to convey an entire experience. They do have the bandwidth to convey “Was it fun?” though.
So what is the conclusion you arrive at? And where can we read the full text of the study, since it is gated.
The last quote you provide is not very informative. I read that the woman had the experience FIRST and was asked AFTERWARDS if she could have made an accurate prediction. From the rest of your post I understood that it was supposed to happen the other way round.
The woman had the experience first and was asked afterwards which type of information would help her make an accurate prediction for both the date she just had, and for another “date”. (Don’t know if these were separate questions.)
I think that’s the right way of doing it. So I withdraw my objection.
Should I delete this post, since the article turns out not to have made the error that I thought it might have made?
I wouldn’t delete a post that has this many comments; I’d add a preface briefly pointing out the error.
I second ciphergoth, don’t delete, add a preface or explanation.
Edit the post.