[Link] You Should Downvote Contrarian Anecdotes
http://thobbs.github.com/blog/2012/06/17/you-should-downvote-anecdotes/
Anecdotal evidence has been shown to have a greater influence on opinion than it logically deserves, most visibly when the anecdote conflicts with the reader’s opinion and when the reader is not highly analytical, even if the anecdotes are accompanying statistical evidence. Though the anecdotes may not totally sway you, they can easily leave you with the sense that the research findings aren’t as conclusive as they claim to be.
The important thing to keep in mind is that this is true in a context where good statistical evidence is available. Rationalist advice tends to degrade over time to its literal meaning. In light of the title it is important to emphasise that someone providing contrary anecdotal evidence against your own or someone else’s anecdotal evidence, obviously shouldn’t be down voted.
I don’t see what that sentence and link have to do with each other.
Konkvistador says quite prominently in it:
He then spends two paragraphs elaborating on it.
It doesn’t follow from this that one should downvote contrarian anecdotes. There is the obvious alternative of de-biasing the readers (e.g., through comments, PMs, posts) and oneself. This is, of course, a monumental and difficult task, but author does not mention any alternatives, which makes one question how much he has considered before settling on his solution.
(Full-texts of linked papers: [1], [2])
Providing de-biasing information like thorough statistics does not eliminate the effects of anecdotes or stories, though. Consider the meta-analysis Winterbottom et al 2008.
(Ref borrowed from my usual fiction-is-bad essay.)
Thanks for the links. I do not have in mind any general de-biasing technique that I would expect to work online (or offline, for that matter).
On de-biasing: a reminder that anyone who does have in mind a de-biasing technique can submit it to CFAR to get $500 (if adopted as an exercise; otherwise $50 if it’s good enough to be tested):
http://wiki.lesswrong.com/wiki/CFAR_Exercise_Prize
Ioannidis showed very clearly, that as a general rule research findings aren’t as conclusive as they claim to be. Research continuely shows that people suffer from overconfidence.
If contrarian anecdotes move you into being a bit more skeptical that’s a good thing.
My ape brain: “Okay, but these findings aren’t as conclusive as they claim to be. One time, I read this study that said animal protein lead to obesity. But my brother said that he ate a ton of animal protein and that it helped him lose weight. And then it turned out the study was wrong! So in my experience it pays to listen to contrarian anecdotes.”
Relevant SMBC.
Actually, our minds and memories are particularly receptive to any narrative. Which can also result in problems, both the narrative fallacy (“Our need to fit a story or pattern to a series of connected or disconnected facts” which leads us to overvalue “facts” embedded in a story format) and undervaluing of statistical and other numerical data. I think that particularly overvaluing contrarian anecdotes is context dependent, since we also overvalue anecdotes we agree with (confirmation bias).
That’s not all bad. Research findings generally aren’t as conclusive as they seem to be, or as people take them to be. And the conclusions that people extrapolate from research findings tend to be unjustified generalizations.
As the years have gone by, the more I’ve learned of statistics and probability, the less I’ve been impressed with statistical tests, and found that they, and unjustified conclusions from them, have a greater influence with some than they logically deserve. There are lies, damned lies, and statistics.
The medical example he uses is a prime one. Treatment T consisting of some dosage schedule D of medicine M on some sample of users with symptom complex C fails to reject the null hypothesis for statistic S over the sample. Therefore it’s a poor decision to try to use medicine M to cure your C.
Just not justified, in so so many ways, and yet you’ll hear it all the time. From my experience, I don’t think too many at Less Wrong need encouragement to discount anecdotes, but a fair number should be encouraged to trust less the conclusions extrapolated from people with tables of numbers and some equations.
I do sometimes depending how the anecdotes are being used.
Hacker News: http://news.ycombinator.com/item?id=4125198
http://en.wikipedia.org/wiki/Elaboration_likelihood_model
I wonder if this is related to near/far or systems one and two.
You call them contrarian anecdotes, I call them “sanity tests” and “smell tests.”
NO, I do not have a study to support my position. I imagine one could be created and done, but one of the reasons you shouldn’t ignore anecdotes is that studies cover a much tinier part of human experience than do anecdotes. Studies of any quality are EXPENSIVE.
Further, I doubt there is a study in the world that wasn’t designed by someone with the intention of confirming their hypothesis, essentially confirming their anecdote. There is a bias to believing only studies, you are limiting yourself to a tiny fraction of human knowledge, and to ideas which were able to garner the funding needed to study them.
Studies are expensive, but they are worth it, I hardly suggest ignoring them in favor of anecdotes. But anecdotes, intuitions, beliefs, these are all what LEAD the people with the resources to study particular hypotheses. One can not gain knowledge optimally without participating in this process, without comparing the anecdotes against the studies to see if the studies really are sufficient to support their conclusions, or whether possibly the conclusions drawn from the studies are too broad, not constrained enough by the richness of the other not-yet-tested hypotheses out there?
As to whether “people” (whoever that means) tend to overweight anecdotes, that is worth considering. And worth correcting in oneself and others if you believe it is happening. But what people? Am I one of those people that overweights anecdotes? I have a PhD in physics, I’m in a sense TRAINED to consider anecdotes against studies, to sift through intuitions to attempt to determine which tiny fraction of them that I can afford to test I will gain the most from testing.
Should people with the proper credentials be allowed to see anecdotes, but have them hidden from the masses? This would be a recipe for authoritarianism, another famous human bias we would do well to avoid. It is also a strawman set up by me, not the OP, so don’t pay too much heed.
But the solution is intelligent discussion and consideration.
And I probably never will buy a Toyota again, not because of someone else’s anecdote but because of my own deplorable experience with Toyota’s lack of response to the squeaky brakes on my 1999 Sienna. I don’t need to pay a premium for reputation if I’m going to get treated like a sucker.
You might benefit from doing some brief research on the “bias blind spot”.
I think I would favor a change to the boards that would require all downvotes to have a comment with them.
(Downvoted because this is a terrible idea. In my view, the function of a downvote is to improve the value of reading the site by means of cheaply, anonymously communicating the information “Someone would like to see fewer comments like this.” If a comment receives multiple downvotes and you’re genuinely confused and curious as to why, it’s reasonable to politely ask for an explanation, but too many comments of the form Why was this downvoted?---downvoters, explain yourselves just hurts the signal-to-noise ratio. Making a policy of explaining all downvotes would be ridiculous; it would generate so much noise that it might be worse than having no karma system at all.)
I downvoted this and the grandparent, in accord to the “I would like to see fewer comments like this” downvoting policy. I do not believe that mwengler benefited from me making this explicit in a comment.
I was incorrectly presuming a somewhat more complex standard than pure incoherent preference. And so I have benefited from seeing the reason made explicit. Thank you.
What about upvotes?