Overall, the use of the term “license” here raises yellow flags for me (see Hero Licensing for the basic reason). It conflates social standing with epistemic standing. The first paragraph here seems a bit confused in other ways too, let me try to break it up into what I see as comparatively crisp distinct claims.
CLAIM: Saying that there are likely flaws in mainstream medical research [...] doesn’t license you to place more credibility in small, poorly performed studies that contradict large, well-performed studies, or in fringe theories that contradict mainstream theories.
CLAIM: Saying that there are likely flaws in mainstream medical research doesn’t license you to discount any specific medical finding unless you have particular reason to believe that finding is false.
The burden of evidence is on the people making claims that constrain our anticipations. We can and should discount all statistical findings in proportion to the evidence that they’re unreliable.
CLAIM: Unless you hold your favorite theory, be it anti-vax, paleo-diet, or whatever, to the same high standard you hold the medical mainstream, every true fact you learn about flaws in medical research makes you stupider.
Statistics isn’t the only sort of evidence. “The same high standards” isn’t necessarily meaningful here, as different kinds of standards are required for different kinds of evidence. There’s also checking a claim against your model of how the world works, and trying things to see whether they produce the claimed effect. The Paleo argument matches my underlying understanding of how the world works, while I’d find the opposite claim pretty counterintuitive. Then I tried eating vaguely Paleo and I felt noticeably better and lost 20 pounds. I have a bunch of friends who report vaguely similar results. Anecdotes like this don’t overrule strong statistical evidence, but that doesn’t mean much when there’s not strong statistical evidence. A little evidence is often better than none, and “standards” may not be the right paradigm here.
It’s nice to be careful not to overgeneralize from personal experience, and it’s also nice to be careful not to overgeneralize from unreplicated underpowered studies, especially when they find something counterintuitive that contradicts my life experience or would be hard for me to check.
Overall, the use of the term “license” here raises yellow flags for me (see Hero Licensing for the basic reason). It conflates social standing with epistemic standing. The first paragraph here seems a bit confused in other ways too, let me try to break it up into what I see as comparatively crisp distinct claims.
CLAIM: Saying that there are likely flaws in mainstream medical research [...] doesn’t license you to place more credibility in small, poorly performed studies that contradict large, well-performed studies, or in fringe theories that contradict mainstream theories.
This seems basically true, and is an application of Beware Isolated Demands for Rigor.
CLAIM: Saying that there are likely flaws in mainstream medical research doesn’t license you to discount any specific medical finding unless you have particular reason to believe that finding is false.
The burden of evidence is on the people making claims that constrain our anticipations. We can and should discount all statistical findings in proportion to the evidence that they’re unreliable.
CLAIM: Unless you hold your favorite theory, be it anti-vax, paleo-diet, or whatever, to the same high standard you hold the medical mainstream, every true fact you learn about flaws in medical research makes you stupider.
Statistics isn’t the only sort of evidence. “The same high standards” isn’t necessarily meaningful here, as different kinds of standards are required for different kinds of evidence. There’s also checking a claim against your model of how the world works, and trying things to see whether they produce the claimed effect. The Paleo argument matches my underlying understanding of how the world works, while I’d find the opposite claim pretty counterintuitive. Then I tried eating vaguely Paleo and I felt noticeably better and lost 20 pounds. I have a bunch of friends who report vaguely similar results. Anecdotes like this don’t overrule strong statistical evidence, but that doesn’t mean much when there’s not strong statistical evidence. A little evidence is often better than none, and “standards” may not be the right paradigm here.
It’s nice to be careful not to overgeneralize from personal experience, and it’s also nice to be careful not to overgeneralize from unreplicated underpowered studies, especially when they find something counterintuitive that contradicts my life experience or would be hard for me to check.