In fact, the study showed fairly substantial improvements in the percentage of patients with depression, high blood pressure, high cholesterol, and high glycated hemoglobin levels (a marker of diabetes). The problem is that the sample size of the study was fairly small, so the results weren’t statistically significant at the 95 percent level.
From a Bayesian perspective, the Oregon results should slightly increase our belief that access to Medicaid produces positive results for diabetes, cholesterol levels, and blood pressure maintenance. It shouldn’t increase our belief much, but if you toss the positive point estimates into the stew of everything we already know, they add slightly to our prior belief that Medicaid is effective.
If this were the only medical study in all of history, then yes, a non-significant result should cause you to update as your quote says. In a world with thousands of studies yearly, you cannot do any such thing, because you’re sure to bias yourself by paying attention to the slightly-positive results you like, and ignore the slightly-negative ones you dislike. (That’s aside from the well-known publication bias where positive results are reported and negative ones aren’t.) If the study had come out with a non-significant negative effect, would comrade Drum have been updating slightly in the direction of “Medicaid is bad”? Hah. This is why we impose the 95% confidence cutoff, which actually is way too low, but that’s another discussion. It prevents us from seeing, or worse, creating, patterns in the noise, which humans are really good at.
The significance cutoff is not a technique of rationality, it is a technique of science, like blinding your results while you’re studying the systematics. It’s something we do because we run on untrusted hardware. Please do not relax your safeguards if a noisy result happens to agree with your opinions! That’s what the safeguards are for!
Then also, please note that Kevin Drum’s prior was not actually “Medicaid will slightly improve these three markers”, it was “Medicaid will drastically reduce mortality”. (See links in discussion with TheOtherDave, below). If you switch your priors around as convenient for claiming support from studies, then of course no study can possibly cause you to update downwards. I would gently suggest that this is not a good epistemic state to occupy.
That is Kevin Drum’s take. Post 1:
Post 2:
If this were the only medical study in all of history, then yes, a non-significant result should cause you to update as your quote says. In a world with thousands of studies yearly, you cannot do any such thing, because you’re sure to bias yourself by paying attention to the slightly-positive results you like, and ignore the slightly-negative ones you dislike. (That’s aside from the well-known publication bias where positive results are reported and negative ones aren’t.) If the study had come out with a non-significant negative effect, would comrade Drum have been updating slightly in the direction of “Medicaid is bad”? Hah. This is why we impose the 95% confidence cutoff, which actually is way too low, but that’s another discussion. It prevents us from seeing, or worse, creating, patterns in the noise, which humans are really good at.
The significance cutoff is not a technique of rationality, it is a technique of science, like blinding your results while you’re studying the systematics. It’s something we do because we run on untrusted hardware. Please do not relax your safeguards if a noisy result happens to agree with your opinions! That’s what the safeguards are for!
Then also, please note that Kevin Drum’s prior was not actually “Medicaid will slightly improve these three markers”, it was “Medicaid will drastically reduce mortality”. (See links in discussion with TheOtherDave, below). If you switch your priors around as convenient for claiming support from studies, then of course no study can possibly cause you to update downwards. I would gently suggest that this is not a good epistemic state to occupy.