Okay, but don’t make the mistake of the guy who says “The mainstream media is all lies—so I’ll only trust what I read on shady Internet conspiracy sites”. Saying that there are likely flaws in mainstream medical research doesn’t license you to discount any specific medical finding unless you have particular reason to believe that finding is false. And it certainly doesn’t license you to place more credibility in small, poorly performed studies that contradict large, well-performed studies, or in fringe theories that contradict mainstream theories. Unless you hold your favorite theory, be it anti-vax, paleo-diet, or whatever, to the same high standard you hold the medical mainstream, every true fact you learn about flaws in medical research makes you stupider.
The study mentioned above looks at exciting cutting-edge research over the past decade. It says that 40% or so was proven wrong. This is good and to the credit of medical science! It means the system is working as it should in retesting things and getting the false stuff out. The basis of science isn’t getting everything right the first time, it’s making sure everyone’s work gets checked and double-checked until only the truth survives. An unreplicated study in almost any area is an intriguing possibility and nothing more; medicine is no exception. If the media makes a big deal about a new study and publishes “VITAMIN B CURES BREAST CANCER!!!” in 72 point font in the newspapers, that is an interesting fact about the media and the people who believe it, but not an interesting fact about medical science.
Good doctors are both conservative and utilitarian. They stick to older, well-proven treatments unless the advantage of a new treatment is so great that it outweighs the uncertainty and risks involved. IMHO the medical consensus has been right on the important things a surprising amount of the time.
I would strongly discourage people from bewaring statins overly much. I don’t see anything by Ioannidis saying the studies surrounding statins are particularly bad. Ioannidis says research is less likely to be true if it has low sample sizes, low effect sizes, bias, and a wide net. There have been several statin trials with sample sizes in the thousands to tens of thousands (see: JUPITER, SSSS, etc.) They’ve found that death rate from heart attacks in people correctly prescribed statin goes down by 30%, which is not at all a small effect size. Many such trials have not been linked to statin manufacturers or anyone with an axe to grind. And because people already know statins are supposed to reduce cholesterol, there is much less of a wide net than if you were to give a bunch of people statins and, say, see if any diseases became less common—the studies had a clearly designated endpoint, which they achieved.
Are there people who suggest the side effects of statins are worse than everyone else thinks? A few, and based off of very little evidence (I believe the idea that statins cause memory dysfunction is based mostly off isolated case reports, and there are only 60 out of many years of hundreds of thousands of people on statins—basically background noise). I haven’t investigated this thoroughly, but the side effects would have to be pretty darned bad and pretty darned robust to stop prescribing a drug with an NNT in the two digits (ie it takes under 100 statin prescriptions to prevent one heart attack), and I treat people trying to exaggerate drug side effects as just as real a failure mode as doctors trying to exaggerate drug benefits, and use just as much caution.
The advice in the third-to-last paragraph, except perhaps the specific singling out of statins, remains excellent.
Overall, the use of the term “license” here raises yellow flags for me (see Hero Licensing for the basic reason). It conflates social standing with epistemic standing. The first paragraph here seems a bit confused in other ways too, let me try to break it up into what I see as comparatively crisp distinct claims.
CLAIM: Saying that there are likely flaws in mainstream medical research [...] doesn’t license you to place more credibility in small, poorly performed studies that contradict large, well-performed studies, or in fringe theories that contradict mainstream theories.
CLAIM: Saying that there are likely flaws in mainstream medical research doesn’t license you to discount any specific medical finding unless you have particular reason to believe that finding is false.
The burden of evidence is on the people making claims that constrain our anticipations. We can and should discount all statistical findings in proportion to the evidence that they’re unreliable.
CLAIM: Unless you hold your favorite theory, be it anti-vax, paleo-diet, or whatever, to the same high standard you hold the medical mainstream, every true fact you learn about flaws in medical research makes you stupider.
Statistics isn’t the only sort of evidence. “The same high standards” isn’t necessarily meaningful here, as different kinds of standards are required for different kinds of evidence. There’s also checking a claim against your model of how the world works, and trying things to see whether they produce the claimed effect. The Paleo argument matches my underlying understanding of how the world works, while I’d find the opposite claim pretty counterintuitive. Then I tried eating vaguely Paleo and I felt noticeably better and lost 20 pounds. I have a bunch of friends who report vaguely similar results. Anecdotes like this don’t overrule strong statistical evidence, but that doesn’t mean much when there’s not strong statistical evidence. A little evidence is often better than none, and “standards” may not be the right paradigm here.
It’s nice to be careful not to overgeneralize from personal experience, and it’s also nice to be careful not to overgeneralize from unreplicated underpowered studies, especially when they find something counterintuitive that contradicts my life experience or would be hard for me to check.
That’s a very good point—medical science being bad does not imply that your favorite contrarian idea is good.
So if I understand you right, the moral that we should draw is that the most reliable treatments are the old ones. A new study constitutes very weak evidence in favor of anything.
I remain skeptical of medicine, but there is a particular kind of emotional/intellectual scam that is practiced by some alternative medicine practitioners, akin to what Yvain cautions.
Particularly when the standard, tested regimen (say chemo + radiation for cancer) has miserable side effects, it’s tempting to go instead for the herbal remedy (or the like).
But until those alternative treatments are subjected to large trials, what we’re trading, basically, is the flawed, but at least somewhat familiar and certainly broad-based approach of medical research for the much murkier world of individual “authorities” and anecdotes. It’s darkly ironic that the same folks who urge me to discount the authority of the medical establishment are eager to have me listen to their “authorities” instead.
The emotional (and generally unacknowledged) part of this scam is that the non-traditional recipe is particularly appealing when the standard remedy is horrible.
You may be right about the statins—part of what spooked me about them was running into a woman whose husband had taken permanent muscle damage from them, which suggested to me that the side effect might not be all that rare.
You mentioned that it was important for them to be correctly prescribed. How common is it for them to not be correctly prescribed?
Rhabdomyolysis, which I think is the kind of severe permanent muscle damage you’re talking about, is well-known enough as a side effect of statins that it’s taught in first year medical school classes. There was one statin that may have had a relatively high (1/2,000 per year) rhabdomyolysis rate and was withdrawn from the market after a couple of years for that reason. The statins currently on the market have about a 1⁄20,000/year rhabdomyolysis rate, which is actually low enough that no one is entirely sure it’s not background noise although no one’s taking any chances. Since they also have a 1+/500/year heart attack prevention rate, they prevent something like 50 heart attacks for each case of rhabdomyolysis they cause, which seems “worth it”.
Muscle damage rates increase by a lot if you take statins with fibrates (another cholesterol lowering drug). I think (not sure) that prescribing these two drugs together is Officially Discouraged, although there might be some leeway in cases of people with crazy high cholesterol. I’ve also heard having grapefruit juice with statins increases the risk (grapefruit juice messes with liver enzymes) but I’m not sure if that is practically important or just random clinical trivia.
As for correct prescription: I am only a student, I haven’t checked the official guidelines, and if you hear otherwise from any doctor trust the doctor and not me—however, as I understand it there is pretty good evidence for giving a statin to people who have already had a cardiovascular event in order to prevent a second one, and much weaker evidence (depending on whose studies and meta-analyses you prefer) for giving it to someone who’s never had a cardiovascular event. Many doctors give them to the latter category anyway just because irreversible side effects are so rare and they would rather be safe than sorry regarding heart attacks; I see some merit in both sides of the argument.
My only evidence is that they’re mostly by large and respected institutions, considered exemplary by the medical community, and that reading a one page summary of them I didn’t come across anything that made me think they weren’t.
Muscle/nerve damage and amnesia as side effects of statins It sounds as though memory problems aren’t routinely monitored, and there’s no way of telling (or at least nothing that gets used) if there’s muscle damage until the patient gets into trouble.
This doesn’t prove that statins aren’t worth the risks.
Okay, but don’t make the mistake of the guy who says “The mainstream media is all lies—so I’ll only trust what I read on shady Internet conspiracy sites”. Saying that there are likely flaws in mainstream medical research doesn’t license you to discount any specific medical finding unless you have particular reason to believe that finding is false. And it certainly doesn’t license you to place more credibility in small, poorly performed studies that contradict large, well-performed studies, or in fringe theories that contradict mainstream theories. Unless you hold your favorite theory, be it anti-vax, paleo-diet, or whatever, to the same high standard you hold the medical mainstream, every true fact you learn about flaws in medical research makes you stupider.
The study mentioned above looks at exciting cutting-edge research over the past decade. It says that 40% or so was proven wrong. This is good and to the credit of medical science! It means the system is working as it should in retesting things and getting the false stuff out. The basis of science isn’t getting everything right the first time, it’s making sure everyone’s work gets checked and double-checked until only the truth survives. An unreplicated study in almost any area is an intriguing possibility and nothing more; medicine is no exception. If the media makes a big deal about a new study and publishes “VITAMIN B CURES BREAST CANCER!!!” in 72 point font in the newspapers, that is an interesting fact about the media and the people who believe it, but not an interesting fact about medical science.
Good doctors are both conservative and utilitarian. They stick to older, well-proven treatments unless the advantage of a new treatment is so great that it outweighs the uncertainty and risks involved. IMHO the medical consensus has been right on the important things a surprising amount of the time.
I would strongly discourage people from bewaring statins overly much. I don’t see anything by Ioannidis saying the studies surrounding statins are particularly bad. Ioannidis says research is less likely to be true if it has low sample sizes, low effect sizes, bias, and a wide net. There have been several statin trials with sample sizes in the thousands to tens of thousands (see: JUPITER, SSSS, etc.) They’ve found that death rate from heart attacks in people correctly prescribed statin goes down by 30%, which is not at all a small effect size. Many such trials have not been linked to statin manufacturers or anyone with an axe to grind. And because people already know statins are supposed to reduce cholesterol, there is much less of a wide net than if you were to give a bunch of people statins and, say, see if any diseases became less common—the studies had a clearly designated endpoint, which they achieved.
Are there people who suggest the side effects of statins are worse than everyone else thinks? A few, and based off of very little evidence (I believe the idea that statins cause memory dysfunction is based mostly off isolated case reports, and there are only 60 out of many years of hundreds of thousands of people on statins—basically background noise). I haven’t investigated this thoroughly, but the side effects would have to be pretty darned bad and pretty darned robust to stop prescribing a drug with an NNT in the two digits (ie it takes under 100 statin prescriptions to prevent one heart attack), and I treat people trying to exaggerate drug side effects as just as real a failure mode as doctors trying to exaggerate drug benefits, and use just as much caution.
The advice in the third-to-last paragraph, except perhaps the specific singling out of statins, remains excellent.
Overall, the use of the term “license” here raises yellow flags for me (see Hero Licensing for the basic reason). It conflates social standing with epistemic standing. The first paragraph here seems a bit confused in other ways too, let me try to break it up into what I see as comparatively crisp distinct claims.
CLAIM: Saying that there are likely flaws in mainstream medical research [...] doesn’t license you to place more credibility in small, poorly performed studies that contradict large, well-performed studies, or in fringe theories that contradict mainstream theories.
This seems basically true, and is an application of Beware Isolated Demands for Rigor.
CLAIM: Saying that there are likely flaws in mainstream medical research doesn’t license you to discount any specific medical finding unless you have particular reason to believe that finding is false.
The burden of evidence is on the people making claims that constrain our anticipations. We can and should discount all statistical findings in proportion to the evidence that they’re unreliable.
CLAIM: Unless you hold your favorite theory, be it anti-vax, paleo-diet, or whatever, to the same high standard you hold the medical mainstream, every true fact you learn about flaws in medical research makes you stupider.
Statistics isn’t the only sort of evidence. “The same high standards” isn’t necessarily meaningful here, as different kinds of standards are required for different kinds of evidence. There’s also checking a claim against your model of how the world works, and trying things to see whether they produce the claimed effect. The Paleo argument matches my underlying understanding of how the world works, while I’d find the opposite claim pretty counterintuitive. Then I tried eating vaguely Paleo and I felt noticeably better and lost 20 pounds. I have a bunch of friends who report vaguely similar results. Anecdotes like this don’t overrule strong statistical evidence, but that doesn’t mean much when there’s not strong statistical evidence. A little evidence is often better than none, and “standards” may not be the right paradigm here.
It’s nice to be careful not to overgeneralize from personal experience, and it’s also nice to be careful not to overgeneralize from unreplicated underpowered studies, especially when they find something counterintuitive that contradicts my life experience or would be hard for me to check.
That’s a very good point—medical science being bad does not imply that your favorite contrarian idea is good.
So if I understand you right, the moral that we should draw is that the most reliable treatments are the old ones. A new study constitutes very weak evidence in favor of anything.
I remain skeptical of medicine, but there is a particular kind of emotional/intellectual scam that is practiced by some alternative medicine practitioners, akin to what Yvain cautions.
Particularly when the standard, tested regimen (say chemo + radiation for cancer) has miserable side effects, it’s tempting to go instead for the herbal remedy (or the like).
But until those alternative treatments are subjected to large trials, what we’re trading, basically, is the flawed, but at least somewhat familiar and certainly broad-based approach of medical research for the much murkier world of individual “authorities” and anecdotes. It’s darkly ironic that the same folks who urge me to discount the authority of the medical establishment are eager to have me listen to their “authorities” instead.
The emotional (and generally unacknowledged) part of this scam is that the non-traditional recipe is particularly appealing when the standard remedy is horrible.
You may be right about the statins—part of what spooked me about them was running into a woman whose husband had taken permanent muscle damage from them, which suggested to me that the side effect might not be all that rare.
You mentioned that it was important for them to be correctly prescribed. How common is it for them to not be correctly prescribed?
Rhabdomyolysis, which I think is the kind of severe permanent muscle damage you’re talking about, is well-known enough as a side effect of statins that it’s taught in first year medical school classes. There was one statin that may have had a relatively high (1/2,000 per year) rhabdomyolysis rate and was withdrawn from the market after a couple of years for that reason. The statins currently on the market have about a 1⁄20,000/year rhabdomyolysis rate, which is actually low enough that no one is entirely sure it’s not background noise although no one’s taking any chances. Since they also have a 1+/500/year heart attack prevention rate, they prevent something like 50 heart attacks for each case of rhabdomyolysis they cause, which seems “worth it”.
Muscle damage rates increase by a lot if you take statins with fibrates (another cholesterol lowering drug). I think (not sure) that prescribing these two drugs together is Officially Discouraged, although there might be some leeway in cases of people with crazy high cholesterol. I’ve also heard having grapefruit juice with statins increases the risk (grapefruit juice messes with liver enzymes) but I’m not sure if that is practically important or just random clinical trivia.
As for correct prescription: I am only a student, I haven’t checked the official guidelines, and if you hear otherwise from any doctor trust the doctor and not me—however, as I understand it there is pretty good evidence for giving a statin to people who have already had a cardiovascular event in order to prevent a second one, and much weaker evidence (depending on whose studies and meta-analyses you prefer) for giving it to someone who’s never had a cardiovascular event. Many doctors give them to the latter category anyway just because irreversible side effects are so rare and they would rather be safe than sorry regarding heart attacks; I see some merit in both sides of the argument.
Thank you for doing the research.
I’ve had a little more time to think—how sure are you that the studies you cite were well-constructed?
My only evidence is that they’re mostly by large and respected institutions, considered exemplary by the medical community, and that reading a one page summary of them I didn’t come across anything that made me think they weren’t.
Muscle/nerve damage and amnesia as side effects of statins It sounds as though memory problems aren’t routinely monitored, and there’s no way of telling (or at least nothing that gets used) if there’s muscle damage until the patient gets into trouble.
This doesn’t prove that statins aren’t worth the risks.