How to prove anything with a review article
Thus the subtitle of this blog posting at PLoS, referencing this article on “Cigarette smoking: an underused tool in high-performance endurance training”. The point being that you can write a review article to argue anything you want, with sufficient cherry-picking and chains of links.
If you are doing actual experiments and making observations or proving theorems, then to a large extent—larger in some sciences than in others—you are constrained by the brute facts. But when writing secondary literature, especially in areas where data is generally fuzzier, it is easy, whether deliberately or not, to write to a bottom line, including findings you like and excluding those you don’t.
Something to bear in mind when reading or writing any review article.
Upvoted. This is a failure LW may have a hard time detecting because of our love of what basically amount to review articles.
Amusing, but one man’s modus ponens is another man’s modus tollens. We know that nicotine has many performance benefits; to jump from a recommendation to use nicotine—which seems perfectly justified to me! - to a recommendation to smoke tobacco is not a matter of cherrypicking but ignoring what is actually being shown.
For that matter, this seems suspiciously like applause lights, with the implicit argument ‘smoking is bad, hence any recommendation to smoke refutes the method or sources used to produce the recommendation’.
Do we actually know that smoking hurts athletic performance on net, or are we just going with the general massive current social prejudice against smoking? After all, gymnasts or ballerinas were notorious for smoking, or so The Simpsons had lead me to believe. And the review’s point that elite athletes rarely smoke could just be due to athletes’ self-image and the aforementioned social pressures (and their failure to embrace nicotine is not evidence, given that the review spent a good chunk of the intro deriding widespread athletic use of proven-worthless & dangerous methods like altitude training).
A family member of mine used to work for Cirque du Soleil. Apparently, those Russian guys who pull off incredible feats of strength and concentration chain smoke while not on stage.
If you want to get athletic benefits from nicotine, the obvious solution is to get it from patches or any other source that doesn’t crud up your lungs. Smokers have perpetually lowered blood oxygen content, something that would be disadvantageous to nearly any athlete.
I don’t know why you’re being downvoted; perhaps for restating part of what I said.
I agree, however I don’t think “smoking is bad” is much of an applause light here. A real applause light would be if the post needlessly referenced things like map and territory.
The review was not written for LW.
Proper quacks even announce that they’re cherry-picking in the title… An Overview of Positive Homeopathy Research (PDF).
As far as I can tell, there is a strong trend, at least in the medical sciences towards more rigorous meta-analyses (eg, the Cochrane Reviews) which to a large extent avoid this problem. I’m not sure to what extent meta-analyses of this type are the norm in other sciences.
I think homeopathy is as silly as you do, but there are reasons to look at only the positive results, and the first step to doing it right is acknowledging what you’re doing, so I don’t fault them for the title.
Consider cases where there are a lot of unknowns on how to do it right. Most studies will find no effect, which is very good evidence that the average researcher can’t make it work and you shouldn’t just ask a random researcher to can treat you. However, if you look at the distribution, you might find that there’s a false positive distribution and a second—an interesting distribution of people that it does consistently work for. Then you can go on to study this separate distribution and see what they’re doing right.
I wonder if just the title of that paper is enough to put its author in the running for an Ig Nobel Prize?
I agree that with the overall notion that you can prove many things (many of which are not true), esp. in the fuzzier sciences. However, I think the article cited is not really a good example of that; it’s way too obvious to be part of the rationalist’s arsenal of amusing observations about this irrational world.