Unfortunately it’s not even that simple. This is really only the tip of a rather gnarly iceberg. The sorts of calculations in the post are a start toward decent interpretation, but only a start.
If I have COVID and take 100 BinaxNOW tests, I will get 68 positive results and 32 negative results. All the positives are correct (0% false positive rate) and all the negatives are incorrect (100% false negative rate).
This is almost certainly false, because the test results aren’t independent. They very likely do reliably test some underlying factor that is correlated with having COVID, such as presence of a particular class of proteins at some concentration in the sample. It is more likely that you would see a much larger or smaller fraction of positive tests since the underlying concentration in multiple samples taken from you is likely to be much more consistent than those taken from a collection of random people infected by COVID. So you can’t even “average it out” by doing lots of tests.
What’s worse, some underlying factors are likely to vary between tested populations and various instances of sample collection and so on. So even knowing the proportion of false positives and negatives they (claimed to!) get in their own test population quite likely won’t be the same as your probability of getting a false positive or negative, so you should allow for even wider variance than the advertised figures because you usually don’t know how closely you match their validation testing profile.
Even worse still, you can’t even multiply the chances of false positives or negatives with prior probabilities from other evidence, because factors related to the other evidence might also co-vary with the probabilities of false positives or negatives. For example, suppose you reduce your evaluated chance of having COVID (moderately) by the fact that you’re not displaying symptoms. Then you lower it more by having a negative test. Oops! Many of these sorts of tests are far more likely to give false negatives in people who are not showing symptoms, so you’ve double-counted some of the same evidence!
These are just a few of the extra pitfalls in interpreting tests, and indeed when interpreting statistical evidence of all types.
I wrote this as a side reference for a deep dive on the BinaxNOW that’s coming shortly, and it’ll dig into the numerous, complex, and important issues affecting BinaxNOW accuracy. Short version: the accuracy varies substantially, largely based on viral load. And you’re correct that repeated tests on the same individual will be strongly correlated.
And you’ve convinced me to change the example you cite: I’d gone with the first person for narrative consistency, but I’m shifting it to prioritize technical accuracy.
Unfortunately it’s not even that simple. This is really only the tip of a rather gnarly iceberg. The sorts of calculations in the post are a start toward decent interpretation, but only a start.
This is almost certainly false, because the test results aren’t independent. They very likely do reliably test some underlying factor that is correlated with having COVID, such as presence of a particular class of proteins at some concentration in the sample. It is more likely that you would see a much larger or smaller fraction of positive tests since the underlying concentration in multiple samples taken from you is likely to be much more consistent than those taken from a collection of random people infected by COVID. So you can’t even “average it out” by doing lots of tests.
What’s worse, some underlying factors are likely to vary between tested populations and various instances of sample collection and so on. So even knowing the proportion of false positives and negatives they (claimed to!) get in their own test population quite likely won’t be the same as your probability of getting a false positive or negative, so you should allow for even wider variance than the advertised figures because you usually don’t know how closely you match their validation testing profile.
Even worse still, you can’t even multiply the chances of false positives or negatives with prior probabilities from other evidence, because factors related to the other evidence might also co-vary with the probabilities of false positives or negatives. For example, suppose you reduce your evaluated chance of having COVID (moderately) by the fact that you’re not displaying symptoms. Then you lower it more by having a negative test. Oops! Many of these sorts of tests are far more likely to give false negatives in people who are not showing symptoms, so you’ve double-counted some of the same evidence!
These are just a few of the extra pitfalls in interpreting tests, and indeed when interpreting statistical evidence of all types.
Yes, those are all excellent points.
I wrote this as a side reference for a deep dive on the BinaxNOW that’s coming shortly, and it’ll dig into the numerous, complex, and important issues affecting BinaxNOW accuracy. Short version: the accuracy varies substantially, largely based on viral load. And you’re correct that repeated tests on the same individual will be strongly correlated.
And you’ve convinced me to change the example you cite: I’d gone with the first person for narrative consistency, but I’m shifting it to prioritize technical accuracy.