I think (hope?) most people already realize NHST is terrible. I would be much more interested in hearing if there were an equally-easy-to-use alternative without any baggage (preferably not requiring priors?)
NHST has been taught as The Method Of Science to lots of students. I remember setting these up explicitly in science class. I expect it will remain in the fabric of any given quantitative field until removed with force.
If you’re right that that’s how science works then that should make you distrustful of science. If they deserve any credibility, scientists must have some process by which they drop bad truth-finding methods instead of repeating them out of blind tradition. Do you believe scientific results?
If they deserve any credibility, scientists must have some process by which they drop bad truth-finding methods instead of repeating them out of blind tradition.
Plenty of otherwise-good science is done based on poor statistics. Keep in mind, there are tons and tons of working scientists, and they’re already pretty busy just trying to understand the content of their fields. Many are likely to view improved statistical methods as an unneeded step in getting a paper published. Others are likely to view overthrowing NHST as a good idea, but not something that they themselves have the time or energy to do. Some might repeat it out of “blind tradition”—but keep in mind that the “blind tradition” is an expensive-to-move Schelling point in a very complex system.
I do expect that serious scientific fields will, eventually, throw out NHST in favor of more fundamentally-sound statistical analyses. But, like any social change, it’ll probably take decades at least.
Do you believe scientific results?
Unconditionally? No, and neither should you. Beliefs don’t work that way.
If a scientific paper gives a fundamentally-sound statistical analysis of the effect it purports to prove, I’ll give it more credence than a paper rejecting the null hypothesis at p < 0.05. On the other hand, a study rejecting the null hypothesis at p < 0.05 is going to provide far more useful information than a small collection of anecdotes, and both are probably better than my personal intuition in a field I have no experience with.
Unconditionally? No, and neither should you. Beliefs don’t work that way.
I should have said, “do you believe any scientific results?”
If a scientific paper gives a fundamentally-sound statistical analysis of the effect it purports to prove, I’ll give it more credence than a paper rejecting the null hypothesis at p < 0.05. On the other hand, a study rejecting the null hypothesis at p < 0.05 is going to provide far more useful information than a small collection of anecdotes, and both are probably better than my personal intuition in a field I have no experience with.
To clarify, I wasn’t saying that maybe you shouldn’t believe scientific results because they use NHST specifically. I meant that if you think that scientists tend to stick with bad methods for decades then NHST probably isn’t the only bad method they’re using.
As you say though, NHST is helpful in many cases even if other methods might be more helpful. So I guess it doesn’t say anything that awful about the way science works.
p<.05 means that the null hypothesis is excluded from the 95% confidence interval. Thus there is no political cost and every p-value recipe is a fragment of an existing confidence interval recipe.
added: also, the maximum likelihood estimate is a single number that is closely related to confidence intervals, but I don’t know if is sufficiently well-known among statistically-ignorant scientists to avoid controversy.
I think (hope?) most people already realize NHST is terrible. I would be much more interested in hearing if there were an equally-easy-to-use alternative without any baggage (preferably not requiring priors?)
NHST has been taught as The Method Of Science to lots of students. I remember setting these up explicitly in science class. I expect it will remain in the fabric of any given quantitative field until removed with force.
If you’re right that that’s how science works then that should make you distrustful of science. If they deserve any credibility, scientists must have some process by which they drop bad truth-finding methods instead of repeating them out of blind tradition. Do you believe scientific results?
Plenty of otherwise-good science is done based on poor statistics. Keep in mind, there are tons and tons of working scientists, and they’re already pretty busy just trying to understand the content of their fields. Many are likely to view improved statistical methods as an unneeded step in getting a paper published. Others are likely to view overthrowing NHST as a good idea, but not something that they themselves have the time or energy to do. Some might repeat it out of “blind tradition”—but keep in mind that the “blind tradition” is an expensive-to-move Schelling point in a very complex system.
I do expect that serious scientific fields will, eventually, throw out NHST in favor of more fundamentally-sound statistical analyses. But, like any social change, it’ll probably take decades at least.
Unconditionally? No, and neither should you. Beliefs don’t work that way.
If a scientific paper gives a fundamentally-sound statistical analysis of the effect it purports to prove, I’ll give it more credence than a paper rejecting the null hypothesis at p < 0.05. On the other hand, a study rejecting the null hypothesis at p < 0.05 is going to provide far more useful information than a small collection of anecdotes, and both are probably better than my personal intuition in a field I have no experience with.
I should have said, “do you believe any scientific results?”
To clarify, I wasn’t saying that maybe you shouldn’t believe scientific results because they use NHST specifically. I meant that if you think that scientists tend to stick with bad methods for decades then NHST probably isn’t the only bad method they’re using.
As you say though, NHST is helpful in many cases even if other methods might be more helpful. So I guess it doesn’t say anything that awful about the way science works.
Confidence intervals.
p<.05 means that the null hypothesis is excluded from the 95% confidence interval. Thus there is no political cost and every p-value recipe is a fragment of an existing confidence interval recipe.
added: also, the maximum likelihood estimate is a single number that is closely related to confidence intervals, but I don’t know if is sufficiently well-known among statistically-ignorant scientists to avoid controversy.