Only one out of 21 obstetricians could estimate the probability that an unborn child had Down syndrome given a positive test
Say the doctor knows false positive/negative rates of the test, and also the overall probability of Down syndrome, but doesn’t know how to combine these into the probability of Down syndrome given a positive test result.
Okay, so to the extent that it’s possible, why doesn’t someone just tell them the results of the Bayesian updating in advance? I assume a doctor is told the false positive and negative rates of a test. But what matters to the doctor is the probability that the patient has the disorder. So instead of telling a doctor, “Here is the probability that a patient with Down syndrome will have a negative test result,” why not just directly say, “When the test is positive, here is the probability of the patient actually having Down syndrome. When the test is negative, here is the probability that the patient has Down syndrome.”
Bayes theorem is a general tool that would let doctors manipulate the information they’re given into the probabilities that they care about. But am I crazy to think that we could circumvent much of their need for Bayes theorem by simply giving them different (not necessarily much more) information?
There are counterpoints to consider. But it seems to me that many examples of Bayesian failure in medicine are analogously simple to the above, and could be as simply fixed. The statistical illiteracy of doctors can be offset so long as there are statistically literate people upstream.
This stops working in the case where some of the people upstream can’t be trusted. Consider the following statement:
“The previous test, if you have a positive result, means that the baby has a 25% chance of having Down syndrome, according to the manufacturer. But my patented test will return a positive result in 99% of cases in which the baby has Down syndrome.”
“False positive rate” and “False negative rate” have strict definitions and presumably it is standard to report these numbers as an outcome of clinical trials. Could we similarly define a rigid term to describe the probability of having a disorder given a positive test result, and require that to be reported right along with false positive rates?
Seems worth an honest try, though it might be too hard to define it in such a way as to forestall weaseling.
The term you are requesting is Positive predictive value and Negative predictive value is the term for not having a disorder given a negative test result.
It also points out that these are not solely dependent on the test, and also require a prevalence percentage.
But that being said, you could require each test to be reported with multiple different prevalence percentages:
For instance, using the above example of Downs Syndrome, you could report the results by using the prevalence of Downs Syndrome at several different given maternal ages. (Since prevalence of Down’s Syndrome is significantly related to maternal age.)
The alternative to giving a doctor positive & negative predictive values for each maternal age is to give false positive & negative rates for the test plus the prevalence rate for each maternal age. Not much difference in terms of the information load.
One concern I didn’t consider before is that many doctors would probably resist reporting PPV’s to their patients because they are currently recommending tests that, if they actually admitted the PPV’s, would look ridiculous! (e.g. breast cancer screening).
Another alternative is to provide doctors with a simple, easy-to-use program called Dr. Bayes. The program would take as input:
the doctor’s initial estimate of the chance the patient has the disorder (taking into account whatever the doctor knows about various risk factors)
the false positive and false negative rates of a test.
The program would spit out the probability of having the disorder given positive and negative test results.
Obviously there are already tools on the internet that will implement Bayes theorem for you. But maybe it could be sold to doctors if the interface were designed specifically for them. I could see a smart person in charge of a hospital telling all the doctors at the hospital to incorporate such a program into their diagnostic procedure.
Failing this, another possibility is to solicit the relevant information from the doctor and then do the math yourself. (Being sure to get the doctor’s prior before any test results are in). Not every doctor would be cooperative...but come to think of it, refusal to give you a number is a good sign that maybe you shouldn’t trust that particular doctor anyway.
Okay, so to the extent that it’s possible, why doesn’t someone just tell them the results of the Bayesian updating in advance?
Because then they would be assuming they had all relevant prior information for that particular patient. They don’t.
For example, age of mother, age of father, their genes, when they’ve lived where, what chemicals they’ve been exposed to, etc., are many factors the manufacturer has no knowledge of, but the doctor might. Naturally, it would be helpful for the company to make an online diagnostic model of all known relevant factors available online, updated as new information comes in, but given the regulatory and legal climate (at least here in the US), something so sensible is likely completely infeasible.
The incidence of the disease may be different for different populations while the test manufacturer may not know where and on which patients the test is going to be used.
Also, serious diseases are often tested multiple times by different tests. What would a Bayes-ignorant doctor do with positives from tests A and B which are accompanied with information: “when test A is positive, the patient has 90% chance of having the syndrome” and “when test B is positive, the patient has 75% chance of having the syndrome”? I’d guess most statistically illiterate doctors would go with the estimate of the test done last.
Say the doctor knows false positive/negative rates of the test, and also the overall probability of Down syndrome, but doesn’t know how to combine these into the probability of Down syndrome given a positive test result.
Okay, so to the extent that it’s possible, why doesn’t someone just tell them the results of the Bayesian updating in advance? I assume a doctor is told the false positive and negative rates of a test. But what matters to the doctor is the probability that the patient has the disorder. So instead of telling a doctor, “Here is the probability that a patient with Down syndrome will have a negative test result,” why not just directly say, “When the test is positive, here is the probability of the patient actually having Down syndrome. When the test is negative, here is the probability that the patient has Down syndrome.”
Bayes theorem is a general tool that would let doctors manipulate the information they’re given into the probabilities that they care about. But am I crazy to think that we could circumvent much of their need for Bayes theorem by simply giving them different (not necessarily much more) information?
There are counterpoints to consider. But it seems to me that many examples of Bayesian failure in medicine are analogously simple to the above, and could be as simply fixed. The statistical illiteracy of doctors can be offset so long as there are statistically literate people upstream.
This stops working in the case where some of the people upstream can’t be trusted. Consider the following statement:
“The previous test, if you have a positive result, means that the baby has a 25% chance of having Down syndrome, according to the manufacturer. But my patented test will return a positive result in 99% of cases in which the baby has Down syndrome.”
“False positive rate” and “False negative rate” have strict definitions and presumably it is standard to report these numbers as an outcome of clinical trials. Could we similarly define a rigid term to describe the probability of having a disorder given a positive test result, and require that to be reported right along with false positive rates?
Seems worth an honest try, though it might be too hard to define it in such a way as to forestall weaseling.
If I understand the following Wikipedia page correctly:
http://en.wikipedia.org/wiki/Positive_predictive_value
The term you are requesting is Positive predictive value and Negative predictive value is the term for not having a disorder given a negative test result.
It also points out that these are not solely dependent on the test, and also require a prevalence percentage.
But that being said, you could require each test to be reported with multiple different prevalence percentages:
For instance, using the above example of Downs Syndrome, you could report the results by using the prevalence of Downs Syndrome at several different given maternal ages. (Since prevalence of Down’s Syndrome is significantly related to maternal age.)
thanks, PPV is exactly what I’m after.
The alternative to giving a doctor positive & negative predictive values for each maternal age is to give false positive & negative rates for the test plus the prevalence rate for each maternal age. Not much difference in terms of the information load.
One concern I didn’t consider before is that many doctors would probably resist reporting PPV’s to their patients because they are currently recommending tests that, if they actually admitted the PPV’s, would look ridiculous! (e.g. breast cancer screening).
Another alternative is to provide doctors with a simple, easy-to-use program called Dr. Bayes. The program would take as input: the doctor’s initial estimate of the chance the patient has the disorder (taking into account whatever the doctor knows about various risk factors) the false positive and false negative rates of a test.
The program would spit out the probability of having the disorder given positive and negative test results.
Obviously there are already tools on the internet that will implement Bayes theorem for you. But maybe it could be sold to doctors if the interface were designed specifically for them. I could see a smart person in charge of a hospital telling all the doctors at the hospital to incorporate such a program into their diagnostic procedure.
Failing this, another possibility is to solicit the relevant information from the doctor and then do the math yourself. (Being sure to get the doctor’s prior before any test results are in). Not every doctor would be cooperative...but come to think of it, refusal to give you a number is a good sign that maybe you shouldn’t trust that particular doctor anyway.
Because then they would be assuming they had all relevant prior information for that particular patient. They don’t.
For example, age of mother, age of father, their genes, when they’ve lived where, what chemicals they’ve been exposed to, etc., are many factors the manufacturer has no knowledge of, but the doctor might. Naturally, it would be helpful for the company to make an online diagnostic model of all known relevant factors available online, updated as new information comes in, but given the regulatory and legal climate (at least here in the US), something so sensible is likely completely infeasible.
The incidence of the disease may be different for different populations while the test manufacturer may not know where and on which patients the test is going to be used.
Also, serious diseases are often tested multiple times by different tests. What would a Bayes-ignorant doctor do with positives from tests A and B which are accompanied with information: “when test A is positive, the patient has 90% chance of having the syndrome” and “when test B is positive, the patient has 75% chance of having the syndrome”? I’d guess most statistically illiterate doctors would go with the estimate of the test done last.