NHST has been taught as The Method Of Science to lots of students. I remember setting these up explicitly in science class. I expect it will remain in the fabric of any given quantitative field until removed with force.
If you’re right that that’s how science works then that should make you distrustful of science. If they deserve any credibility, scientists must have some process by which they drop bad truth-finding methods instead of repeating them out of blind tradition. Do you believe scientific results?
If they deserve any credibility, scientists must have some process by which they drop bad truth-finding methods instead of repeating them out of blind tradition.
Plenty of otherwise-good science is done based on poor statistics. Keep in mind, there are tons and tons of working scientists, and they’re already pretty busy just trying to understand the content of their fields. Many are likely to view improved statistical methods as an unneeded step in getting a paper published. Others are likely to view overthrowing NHST as a good idea, but not something that they themselves have the time or energy to do. Some might repeat it out of “blind tradition”—but keep in mind that the “blind tradition” is an expensive-to-move Schelling point in a very complex system.
I do expect that serious scientific fields will, eventually, throw out NHST in favor of more fundamentally-sound statistical analyses. But, like any social change, it’ll probably take decades at least.
Do you believe scientific results?
Unconditionally? No, and neither should you. Beliefs don’t work that way.
If a scientific paper gives a fundamentally-sound statistical analysis of the effect it purports to prove, I’ll give it more credence than a paper rejecting the null hypothesis at p < 0.05. On the other hand, a study rejecting the null hypothesis at p < 0.05 is going to provide far more useful information than a small collection of anecdotes, and both are probably better than my personal intuition in a field I have no experience with.
Unconditionally? No, and neither should you. Beliefs don’t work that way.
I should have said, “do you believe any scientific results?”
If a scientific paper gives a fundamentally-sound statistical analysis of the effect it purports to prove, I’ll give it more credence than a paper rejecting the null hypothesis at p < 0.05. On the other hand, a study rejecting the null hypothesis at p < 0.05 is going to provide far more useful information than a small collection of anecdotes, and both are probably better than my personal intuition in a field I have no experience with.
To clarify, I wasn’t saying that maybe you shouldn’t believe scientific results because they use NHST specifically. I meant that if you think that scientists tend to stick with bad methods for decades then NHST probably isn’t the only bad method they’re using.
As you say though, NHST is helpful in many cases even if other methods might be more helpful. So I guess it doesn’t say anything that awful about the way science works.
NHST has been taught as The Method Of Science to lots of students. I remember setting these up explicitly in science class. I expect it will remain in the fabric of any given quantitative field until removed with force.
If you’re right that that’s how science works then that should make you distrustful of science. If they deserve any credibility, scientists must have some process by which they drop bad truth-finding methods instead of repeating them out of blind tradition. Do you believe scientific results?
Plenty of otherwise-good science is done based on poor statistics. Keep in mind, there are tons and tons of working scientists, and they’re already pretty busy just trying to understand the content of their fields. Many are likely to view improved statistical methods as an unneeded step in getting a paper published. Others are likely to view overthrowing NHST as a good idea, but not something that they themselves have the time or energy to do. Some might repeat it out of “blind tradition”—but keep in mind that the “blind tradition” is an expensive-to-move Schelling point in a very complex system.
I do expect that serious scientific fields will, eventually, throw out NHST in favor of more fundamentally-sound statistical analyses. But, like any social change, it’ll probably take decades at least.
Unconditionally? No, and neither should you. Beliefs don’t work that way.
If a scientific paper gives a fundamentally-sound statistical analysis of the effect it purports to prove, I’ll give it more credence than a paper rejecting the null hypothesis at p < 0.05. On the other hand, a study rejecting the null hypothesis at p < 0.05 is going to provide far more useful information than a small collection of anecdotes, and both are probably better than my personal intuition in a field I have no experience with.
I should have said, “do you believe any scientific results?”
To clarify, I wasn’t saying that maybe you shouldn’t believe scientific results because they use NHST specifically. I meant that if you think that scientists tend to stick with bad methods for decades then NHST probably isn’t the only bad method they’re using.
As you say though, NHST is helpful in many cases even if other methods might be more helpful. So I guess it doesn’t say anything that awful about the way science works.