Is “my grandfather is still alive, in no pain and walking eight miles a day, when six months ago he was given two months to live” stronger evidence for vitamin C effectivity than a reviewed paper saying “we have conducted a study on 1000 patients with terminal cancer; the survival rate in the group treated with large doses of vitamin C was not greater than the survival rate in the control group”? If so, why?
It would depend on the methodology used. I have seen enough examples of horribly bad—not to say utterly fraudulent—science in medical journals that I would actually take publication in a medical journal of a single, unreplicated, study as being slight evidence against the conclusion it comes to.
(As an example, the Mayo clinic published a study with precisely those results, claiming to be ‘unable to replicate’ a previous experiment, in the early 80s. Except that where the experiment they were trying to ‘replicate’ had used intravenous doses, they used oral ones. And used a lower dose. And spaced the doses differently. And ended the trial after a much shorter period.)
So my immediate conclusion would be “No they didn’t” if I saw that result from a single paper. Because when you’ve seen people in agony, dying, and you’ve seen them walking around and healthy a couple of months later, and you see that happen repeatably, then that is very strong evidence. And something you can test yourself is always better than taking someone else’s word for it.
However, if that study were replicated, independently, and had no obviously cretinous methodological flaws upon inspection, then it would be strong evidence. But if something I don’t directly observe myself contradicts my own observations, then I will always put my own observations ahead of those of someone else.
Is “my grandfather is still alive, in no pain and walking eight miles a day, when six months ago he was given two months to live” stronger evidence for vitamin C effectivity than a reviewed paper saying “we have conducted a study on 1000 patients with terminal cancer; the survival rate in the group treated with large doses of vitamin C was not greater than the survival rate in the control group”? If so, why?
It would depend on the methodology used. I have seen enough examples of horribly bad—not to say utterly fraudulent—science in medical journals that I would actually take publication in a medical journal of a single, unreplicated, study as being slight evidence against the conclusion it comes to. (As an example, the Mayo clinic published a study with precisely those results, claiming to be ‘unable to replicate’ a previous experiment, in the early 80s. Except that where the experiment they were trying to ‘replicate’ had used intravenous doses, they used oral ones. And used a lower dose. And spaced the doses differently. And ended the trial after a much shorter period.)
So my immediate conclusion would be “No they didn’t” if I saw that result from a single paper. Because when you’ve seen people in agony, dying, and you’ve seen them walking around and healthy a couple of months later, and you see that happen repeatably, then that is very strong evidence. And something you can test yourself is always better than taking someone else’s word for it.
However, if that study were replicated, independently, and had no obviously cretinous methodological flaws upon inspection, then it would be strong evidence. But if something I don’t directly observe myself contradicts my own observations, then I will always put my own observations ahead of those of someone else.