If this were the only medical study in all of history, then yes, a non-significant result should cause you to update as your quote says. In a world with thousands of studies yearly, you cannot do any such thing, because you’re sure to bias yourself by paying attention to the slightly-positive results you like, and ignore the slightly-negative ones you dislike. (That’s aside from the well-known publication bias where positive results are reported and negative ones aren’t.) If the study had come out with a non-significant negative effect, would comrade Drum have been updating slightly in the direction of “Medicaid is bad”? Hah. This is why we impose the 95% confidence cutoff, which actually is way too low, but that’s another discussion. It prevents us from seeing, or worse, creating, patterns in the noise, which humans are really good at.
The significance cutoff is not a technique of rationality, it is a technique of science, like blinding your results while you’re studying the systematics. It’s something we do because we run on untrusted hardware. Please do not relax your safeguards if a noisy result happens to agree with your opinions! That’s what the safeguards are for!
Then also, please note that Kevin Drum’s prior was not actually “Medicaid will slightly improve these three markers”, it was “Medicaid will drastically reduce mortality”. (See links in discussion with TheOtherDave, below). If you switch your priors around as convenient for claiming support from studies, then of course no study can possibly cause you to update downwards. I would gently suggest that this is not a good epistemic state to occupy.
If this were the only medical study in all of history, then yes, a non-significant result should cause you to update as your quote says. In a world with thousands of studies yearly, you cannot do any such thing, because you’re sure to bias yourself by paying attention to the slightly-positive results you like, and ignore the slightly-negative ones you dislike. (That’s aside from the well-known publication bias where positive results are reported and negative ones aren’t.) If the study had come out with a non-significant negative effect, would comrade Drum have been updating slightly in the direction of “Medicaid is bad”? Hah. This is why we impose the 95% confidence cutoff, which actually is way too low, but that’s another discussion. It prevents us from seeing, or worse, creating, patterns in the noise, which humans are really good at.
The significance cutoff is not a technique of rationality, it is a technique of science, like blinding your results while you’re studying the systematics. It’s something we do because we run on untrusted hardware. Please do not relax your safeguards if a noisy result happens to agree with your opinions! That’s what the safeguards are for!
Then also, please note that Kevin Drum’s prior was not actually “Medicaid will slightly improve these three markers”, it was “Medicaid will drastically reduce mortality”. (See links in discussion with TheOtherDave, below). If you switch your priors around as convenient for claiming support from studies, then of course no study can possibly cause you to update downwards. I would gently suggest that this is not a good epistemic state to occupy.