I currently work as a doctor near Brisbane, Australia. Right now, I’m pre-specialty training, and deciding what I want to do with the rest of my career. Possible candidates, outside of continuing in clinical medicine, are jobs in operations or research at EA-aligned Global Health and Wellbeing organisations, and I’m excited about opportunities in these fields.
Joseph Pusey
Karma: 4
I don’t find the exhortation to abandon per-protocol analysis very compelling without even a basic attempt to understand why educated people acting in good faith would be using it. I’m not a medical research expert, but I can think of at least a few situations off the top off my head where per-protocol studies are useful.
A per-protocol study which showed significant benefit to those who complied with a treatment regimen, but a high rate of drop outs (which in intent-to-treat analysis would be clumped together) would be useful in differentiating between an effective but poorly-tolerated treatment and an ineffective one. This could be the difference between abandoning a potential treatment and researching ways to make it more tolerable
Per-protocol studies give us something to rely on when speaking to highly-motivated patients who are willing and able to stick to demanding regimens and can therefore experience the benefit which we wouldn’t have noticed in an intent-to-treat study
Per-protocol studies can help to establish proof-of-concept for a new medical treatment. It’s useful to demonstrate that a new treatment works under ideal conditions, even if that knowledge doesn’t generalise to real-world populations yet
I agree that it’s useful for papers to contain both per-protocol and intention-to-treat analyses, but your claim here (that no one should ever conduct per-protocol analysis) seems far too strong to me. I’m not sure why you’re so keen to cut off a very important methodology just because there’s a theoretical risk that people who don’t read the details of a study might draw the wrong conclusions from it.
Thanks for your kind, encouraging, and thought-provoking comment Willa :)
Definitely the ideal would have been to write this earlier on- the post itself has been on my list to write for a long time, which probably didn’t help. I like the idea of having some kind of objective comparison method- the ideal would be some kind of Rationality score, but I don’t know if that kind of thing does/could ever exist, or would even reflect the breadth of change one is likely to experience enough to make it even vaguely useful.
I think “dark night of the soul” works pretty well as a descriptor of the experience we’re talking about, although to me it conjures some images of being either guilty or having to make a difficult choice, rather than necessarily specifically having your cognitive foundations shaken. Whatever we call it, I would- as you suggested- be interested to hear how other’s dealt with it, and how they managed to fulfil their responsibilities in other areas of life when at times everything else can suddenly seem quite unimportant.
Your ideas for avoiding our Dark Night sound reasonable and it comforts me that you seem to be a lot further on in your “thought journey” and still find solace somewhere; I guess my worry would be that lifeboats might not be enough for me to retain enough functioning for the rest of my life, and I would like a reasonably solid intellectual terra firma to act from. I think your idea about self-talking with very concrete, predictable things is likely to help, and I have heard of such things being used for anxiety attacks and the like.
I’m interested to hear what your “snapshot” looks like- even if you’re not at the beginning of your journey, it’s probably worth doing both for yourself and for other travellers. And thanks again for your encouraging and thoughtful reply- lots to consider!
Do you have much evidence of a time when medical researchers have been misled by a per-protocol analysis into advancing a treatment which, were it analysed using an intention-to-treat analysis, would not have been taken forward?
As one of I’m sure many good examples of the utility of per-protocol analysis; this study, looking at whether routine screening with colonoscopy reduces the risk of colorectal cancer. In the intention-to-treat analysis (examining those who were invited to be screened), the difference in death rate wasn’t significant. In the per-protocol analysis (examining those who actually took part in screening), the difference was statistically significant.
By banning per-protocol analysis in the way you’re suggesting, there would have been no way of knowing whether it was worth investing more in getting people to actually turn up to screening- the only conclusion of the trial would have been “inviting people to get screened doesn’t significantly reduce their risk of death”. Thanks to per-protocol analysis, we can say “actually, the people who turn up DO die less often- so if we can get people to respond to our invitations, we can reduce the risk of death”.
I still feel that, so far, you have identified what per-protocol analysis is, decided independently that there is an enormous risk of researchers misinterpreting this fairly basic concept, and then making incorrect decisions based on this misunderstanding. I don’t think there’s much evidence to suggest that this potential harm is real, and there’s lots of evidence (both from this specific case study and from the fact that thousands of legitimate medical researchers obviously find it useful) to think that there’s benefit.
If your claim is actually that, without understanding the difference between per-protocol and intention-to-treat analyses, someone superficially reading a paper might misunderstand the conclusions, then I agree it’s technically possible, but I don’t think that’s likely or significant enough to call it “medical malpractice”.