Is it really helpful to insist that the shorthand be expanded on to even more speculative criticisms, or are we just letting the perfect be the enemy of the better here?
Part of the point I hoped to make was that “raising the sanity waterline” would be well served by better awareness of the processes of scientific inference—statistics, experimental design and so on. More people should know about regression to the mean, confounding, biases, unblinding, file drawer effects - specific criticisms.
As a specific example, take the Blackwell study which people elsewhere in this thread have pointed me to, supposedly showing that “chemically inert pills of different colors modulate vigilance”. (A claim counter to everyday experience, in which people take coffee to stay awake, rather than eat (red) strawberries.) I hope you’ll agree that “it’s the placebo effect” isn’t an appropriate shorthand to criticize that particular study.
In this case, “the placebo effect” acts as a semantic stopsign—it kills what legitimate curiosity people should have about this study, given that:
it was conducted in 1972 but appears not to have been replicated since
it fails to detail outcome measures, effect sizes, or significance levels
the sample is highly susceptible to selection bias (medical students)
the sample is highly susceptible to pressure (teacher is out to show something, students know it)
It’s very hard from the abstract to know what to make of the study, a fulltext can’t be found easily, yet people are citing this one “source” all over the place (and in some cases just making the bare claim without even bothering to cite a source). I’ve come to recognize these as red flags.
A claim counter to everyday experience, in which people take coffee to stay awake, rather than eat (red) strawberries.
At least one of us is confused. What conflict do you see between the following two propositions? (1) Eating an otherwise inert red thing can make you more alert. (2) Drinking coffee does more to make you alert than eating a red strawberry does.
(Even if there were a conflict between those, I’d have a problem with what you said, since it could be true that (2′) drinking coffee doesn’t really wake you up more than eating strawberries but (3) people drink coffee anyway, e.g. by habit or because it’s cheaper or because strawberries are more fattening or something. But my main objection is that I don’t see any conflict between 1 and 2.)
“it’s the placebo effect” isn’t an appropriate shorthand to criticize that particular study.
No, it isn’t, but what’s meant to follow from that? I don’t think anyone’s claiming that “it’s the placebo effect” is some kind of universally-insightful response to any observation that involves possible placebos. (Your example even seems to have the wrong sign, as it were; someone inclined to overuse “placebo effect” as an explanation is surely more likely to be defending the Blackwell study than attacking it, and indeed you go on to suggest that those people would fail to criticize the study, not that they would criticize it in an unhelpful way.)
Possibly me. I’m provisionally retracting that; my reasoning was that if eating red things, drinking from red cups etc. reliably increased alertness someone would have noticed and we would be exploiting this effect, not looking for it within the restricted context of eating a pill. However, I’m now remembering that there is just such a claim, called the “red room effect”, which I have no particular reason to disbelieve.
Er… Teaching people about placebo effect is raising the sanity waterline, or do you think >50% of the population knows it and why it is relevant, and at all discounts studies based on it? (I’m pretty sure they don’t, since this is a fine point of randomized studies, while most people read credulously just regular correlative studies and certainly don’t appreciate any specifics of the evidence hierarchy!)
So, you are letting the perfect be the enemy of the better: in objecting that a valid criticism can be even more precisely specified. Thanks for finally being clear about it; now I can downvote the post with a clear conscience.
You can teach people about placebos (or about control responses, or about proper controls in general) without needing to perpetuate the errors stemming from a poorly-named “placebo effect” and the hidden inferences that come with the term “effect”.
Part of the point I hoped to make was that “raising the sanity waterline” would be well served by better awareness of the processes of scientific inference—statistics, experimental design and so on. More people should know about regression to the mean, confounding, biases, unblinding, file drawer effects - specific criticisms.
As a specific example, take the Blackwell study which people elsewhere in this thread have pointed me to, supposedly showing that “chemically inert pills of different colors modulate vigilance”. (A claim counter to everyday experience, in which people take coffee to stay awake, rather than eat (red) strawberries.) I hope you’ll agree that “it’s the placebo effect” isn’t an appropriate shorthand to criticize that particular study.
In this case, “the placebo effect” acts as a semantic stopsign—it kills what legitimate curiosity people should have about this study, given that:
it was conducted in 1972 but appears not to have been replicated since
it fails to detail outcome measures, effect sizes, or significance levels
the sample is highly susceptible to selection bias (medical students)
the sample is highly susceptible to pressure (teacher is out to show something, students know it)
It’s very hard from the abstract to know what to make of the study, a fulltext can’t be found easily, yet people are citing this one “source” all over the place (and in some cases just making the bare claim without even bothering to cite a source). I’ve come to recognize these as red flags.
At least one of us is confused. What conflict do you see between the following two propositions? (1) Eating an otherwise inert red thing can make you more alert. (2) Drinking coffee does more to make you alert than eating a red strawberry does.
(Even if there were a conflict between those, I’d have a problem with what you said, since it could be true that (2′) drinking coffee doesn’t really wake you up more than eating strawberries but (3) people drink coffee anyway, e.g. by habit or because it’s cheaper or because strawberries are more fattening or something. But my main objection is that I don’t see any conflict between 1 and 2.)
No, it isn’t, but what’s meant to follow from that? I don’t think anyone’s claiming that “it’s the placebo effect” is some kind of universally-insightful response to any observation that involves possible placebos. (Your example even seems to have the wrong sign, as it were; someone inclined to overuse “placebo effect” as an explanation is surely more likely to be defending the Blackwell study than attacking it, and indeed you go on to suggest that those people would fail to criticize the study, not that they would criticize it in an unhelpful way.)
Possibly me. I’m provisionally retracting that; my reasoning was that if eating red things, drinking from red cups etc. reliably increased alertness someone would have noticed and we would be exploiting this effect, not looking for it within the restricted context of eating a pill. However, I’m now remembering that there is just such a claim, called the “red room effect”, which I have no particular reason to disbelieve.
Er… Teaching people about placebo effect is raising the sanity waterline, or do you think >50% of the population knows it and why it is relevant, and at all discounts studies based on it? (I’m pretty sure they don’t, since this is a fine point of randomized studies, while most people read credulously just regular correlative studies and certainly don’t appreciate any specifics of the evidence hierarchy!)
So, you are letting the perfect be the enemy of the better: in objecting that a valid criticism can be even more precisely specified. Thanks for finally being clear about it; now I can downvote the post with a clear conscience.
You can teach people about placebos (or about control responses, or about proper controls in general) without needing to perpetuate the errors stemming from a poorly-named “placebo effect” and the hidden inferences that come with the term “effect”.