Check out the title: abuse of frequentist statistics. Yes, at the end, I argue from a Bayesian perspective, but you don’t have to be a Bayesian to see the structural problems with frequentist statistics as currently taught to and practiced by working scientists.
I agree that frequentist statistics are often poorly taught and understood, and that this holds however you like to do your statistics. Still, the main post feels to me like a sales pitch for Bayes brand chainsaws that’s trying to scare me off Neyman-Pearson chainsaws by pointing out how often people using Neyman-Pearson chainsaws accidentally cut off a limb with them. (I am aware that I may be the only reader who feels this way about the post.)
(Does the fact that when I saw the sample size the word “underpowered” instantly jumped into my head count as evidence that I am competent?)
Yes, but it is not sufficient evidence to reject the null hypothesis of incompetence at the 0.05 significance level. (I keed, I keed.)
Still, the main post feels to me like a sales pitch...
It’s a fair point; I’m not exactly attacking the strongest representative of frequentist statistical practice. My only defense is that this actually happened, so it makes a good case study.
I assert that it is evidence in my concluding paragraph, but it’s true that I don’t give an actual argument. Whether one counts it as evidence would seem to depend on the causal assumptions one makes about the teaching and practice of statistics.
The critique of frequentist statistics, as I understand it—and I don’t think I do—is that frequentists like to count things, and trust that having large sample sizes will take care of biases for them. Therefore, a case in which frequentist statistics co-occurs with bad results counts against use of frequentist statistics, and you don’t have to worry about why the results were bad.
The whole Bayesian vs. frequentist argument seems a little silly to me. It’s like arguing that screws are better than nails. It’s true that, for any particular individual joint you wish to connect, a screw will probably connect it more securely and reversibly than a nail. That doesn’t mean there’s no use for nails.
I agree that frequentist statistics are often poorly taught and understood, and that this holds however you like to do your statistics. Still, the main post feels to me like a sales pitch for Bayes brand chainsaws that’s trying to scare me off Neyman-Pearson chainsaws by pointing out how often people using Neyman-Pearson chainsaws accidentally cut off a limb with them. (I am aware that I may be the only reader who feels this way about the post.)
Yes, but it is not sufficient evidence to reject the null hypothesis of incompetence at the 0.05 significance level. (I keed, I keed.)
I get that impression a lot around here
It’s a fair point; I’m not exactly attacking the strongest representative of frequentist statistical practice. My only defense is that this actually happened, so it makes a good case study.
That’s true, and having been reminded of that, I think I may have been unduly pedantic about a fine detail at the expense of the main point.
It’s a good case study, but it’s not evidence of a problem with frequentist statistics.
I assert that it is evidence in my concluding paragraph, but it’s true that I don’t give an actual argument. Whether one counts it as evidence would seem to depend on the causal assumptions one makes about the teaching and practice of statistics.
Perhaps it’s frequentist evidence against frequentist statistics.
I think this is just a glib rejoinder, but if there’s a deeper thought there, I’d be interested to hear it.
The critique of frequentist statistics, as I understand it—and I don’t think I do—is that frequentists like to count things, and trust that having large sample sizes will take care of biases for them. Therefore, a case in which frequentist statistics co-occurs with bad results counts against use of frequentist statistics, and you don’t have to worry about why the results were bad.
The whole Bayesian vs. frequentist argument seems a little silly to me. It’s like arguing that screws are better than nails. It’s true that, for any particular individual joint you wish to connect, a screw will probably connect it more securely and reversibly than a nail. That doesn’t mean there’s no use for nails.