Thanks, I agree that I should better understand why so many medical researchers do this. And I can definitely imagine situations where it helps: small sample (so ITT is under powered), useful treatment (so we’re worried about false negatives), low chance of bias (compliance isn’t correlated with outcomes). My belief is that these three factors don’t align enough to justify per protocol analysis. It’s way more likely to lead you to false cures. It’s hard to test this empirically though so if you had examples of treatments that got correctly assessed and advanced with a per protocol analysis I’d be very interested!
My belief is that these three factors don’t align enough to justify per protocol analysis. It’s way more likely to lead you to false cures
Do you have much evidence of a time when medical researchers have been misled by a per-protocol analysis into advancing a treatment which, were it analysed using an intention-to-treat analysis, would not have been taken forward?
As one of I’m sure many good examples of the utility of per-protocol analysis; this study, looking at whether routine screening with colonoscopy reduces the risk of colorectal cancer. In the intention-to-treat analysis (examining those who were invited to be screened), the difference in death rate wasn’t significant. In the per-protocol analysis (examining those who actually took part in screening), the difference was statistically significant.
By banning per-protocol analysis in the way you’re suggesting, there would have been no way of knowing whether it was worth investing more in getting people to actually turn up to screening- the only conclusion of the trial would have been “inviting people to get screened doesn’t significantly reduce their risk of death”. Thanks to per-protocol analysis, we can say “actually, the people who turn up DO die less often- so if we can get people to respond to our invitations, we can reduce the risk of death”.
I still feel that, so far, you have identified what per-protocol analysis is, decided independently that there is an enormous risk of researchers misinterpreting this fairly basic concept, and then making incorrect decisions based on this misunderstanding. I don’t think there’s much evidence to suggest that this potential harm is real, and there’s lots of evidence (both from this specific case study and from the fact that thousands of legitimate medical researchers obviously find it useful) to think that there’s benefit.
If your claim is actually that, without understanding the difference between per-protocol and intention-to-treat analyses, someone superficially reading a paper might misunderstand the conclusions, then I agree it’s technically possible, but I don’t think that’s likely or significant enough to call it “medical malpractice”.
Thanks, I agree that I should better understand why so many medical researchers do this. And I can definitely imagine situations where it helps: small sample (so ITT is under powered), useful treatment (so we’re worried about false negatives), low chance of bias (compliance isn’t correlated with outcomes). My belief is that these three factors don’t align enough to justify per protocol analysis. It’s way more likely to lead you to false cures. It’s hard to test this empirically though so if you had examples of treatments that got correctly assessed and advanced with a per protocol analysis I’d be very interested!
Do you have much evidence of a time when medical researchers have been misled by a per-protocol analysis into advancing a treatment which, were it analysed using an intention-to-treat analysis, would not have been taken forward?
As one of I’m sure many good examples of the utility of per-protocol analysis; this study, looking at whether routine screening with colonoscopy reduces the risk of colorectal cancer. In the intention-to-treat analysis (examining those who were invited to be screened), the difference in death rate wasn’t significant. In the per-protocol analysis (examining those who actually took part in screening), the difference was statistically significant.
By banning per-protocol analysis in the way you’re suggesting, there would have been no way of knowing whether it was worth investing more in getting people to actually turn up to screening- the only conclusion of the trial would have been “inviting people to get screened doesn’t significantly reduce their risk of death”. Thanks to per-protocol analysis, we can say “actually, the people who turn up DO die less often- so if we can get people to respond to our invitations, we can reduce the risk of death”.
I still feel that, so far, you have identified what per-protocol analysis is, decided independently that there is an enormous risk of researchers misinterpreting this fairly basic concept, and then making incorrect decisions based on this misunderstanding. I don’t think there’s much evidence to suggest that this potential harm is real, and there’s lots of evidence (both from this specific case study and from the fact that thousands of legitimate medical researchers obviously find it useful) to think that there’s benefit.
If your claim is actually that, without understanding the difference between per-protocol and intention-to-treat analyses, someone superficially reading a paper might misunderstand the conclusions, then I agree it’s technically possible, but I don’t think that’s likely or significant enough to call it “medical malpractice”.