I am skeptical of the teaching solution section under 2), relative to institutional shifts (favoring confirmatory vs exploratory studies, etc). Section 3 could also bear mention of some of the many ways of abusing Bayesian statistical analyses (e.g. reporting results based on gerrymandered priors, selecting which likelihood ratio to highlight in the abstract and get media attention for, etc). Cosma Shalizi would have a lot to say about it.
I do like the spirit of the post, but it comes across a bit boosterish.
Section 3 could also bear mention of some of the many ways of abusing Bayesian statistical analyses
On this note, I predict that if Bayesian statistical analyses ever displaced NHST as the mainstream standard, they would be misused about as much as NHST.
Currently there’s a selection bias: NHST is much more widely taught than Bayesian analyses, so NHST users are much more likely to be lowest common demoninator crank-turners who don’t really understand statistics generally. By contrast, if you’ve managed to find out how to do Bayesian inference, you’re probably better at statistics than the average researcher and therefore less likely to screw up whatever analysis you choose to do. If every researcher were taught Bayesian inference this would no longer be true.
Still, I think Bayesian methods are superior enough that the net benefit of that would be positive. (Also, proper Bayesian training would also cover how to construct ignorance priors, and I suspect nefariously chosen priors would be easier to spot than nefarious frequentist mistreatment of data.)
Bayesian methods are better in a number of ways, but ignorant people using a better tool won’t necessarily get better results. I don’t think the net effect of a mass switch to Bayesian methods would be negative, but I do think it’d be very small unless it involved raising the general statistical competence of scientists.
Even when Bayesian methods get so commonplace that they could be used just by pushing a button in SPSS, researchers will still have many tricks at their disposal to skew their conclusions. Not bothering to publish contrary data, only publishing subgroup analyses that show a desired result, ruling out inconvenient data points as “outliers”, wilful misinterpretation of past work, failing to correct for doing multiple statistical tests (and this can be an issue with Bayesian t-tests, like those in the Wagenmakers et al. reanalysis lukeprog linked above), and so on.
As a biologist, I can say that most statistical errors are just that: errors. They are not tricks. If researchers understand the statistics that they are using, a lot of these problems will go away.
A person has to learn a hell of a lot before they can do molecular biology research, and statistics happens to be fairly low on the priority list for most molecular biologists. In many situations we are able to get around the statistical complexities by generating data with very little noise.
ISTM a large benefit of commonplace Bayes would be that competent statisticians could do actually meaningful meta-analyses...? Which would counteract widespread statistical ineptitude to a significant extent...?
I’m not sure it’d make much difference. From reading & skimming meta-analyses myself I’ve inferred that the main speedbumps with doing them are problems with raw data themselves or a lack of access to raw data. Whether the data were originally summarized using NHST/frequentist methods or Bayesian methods makes a lot less difference.
Edit to add: when I say “problems with raw data themselves” I don’t necessarily mean erroneous data; a problem can be as mundane as the sample/dataset not meeting the meta-analyst’s requirements (e.g. if the sample were unrepresentative, or the dataset didn’t contain a set of additional moderator variables).
I don’t think the net effect of a mass switch to Bayesian methods would be negative, but I do think it’d be very small unless it involved raising the general statistical competence of scientists.
I think that teaching Bayesian methods would itself raise the general statistical competence of scientists as a side effect, among other things because the meaning of p-values is seriously counter-intuitive (so more scientists would actually grok Bayesian statistics in such a world than actually grok frequentist statistics right now).
You could well be right. I’m pessimistic about this because I remember seeing lots of people at school & university recoiling from any statistical topic more advanced than calculating means and drawing histograms. If they were being taught about conjugate priors & hyperparameters I’d expect them to react as unenthusiastically as if they were being taught about confidence levels and maximum likelihood. But I don’t have any rock solid evidence for that hunch.
I am skeptical of the teaching solution section under 2), relative to institutional shifts (favoring confirmatory vs exploratory studies, etc). Section 3 could also bear mention of some of the many ways of abusing Bayesian statistical analyses (e.g. reporting results based on gerrymandered priors, selecting which likelihood ratio to highlight in the abstract and get media attention for, etc). Cosma Shalizi would have a lot to say about it.
I do like the spirit of the post, but it comes across a bit boosterish.
On this note, I predict that if Bayesian statistical analyses ever displaced NHST as the mainstream standard, they would be misused about as much as NHST.
Currently there’s a selection bias: NHST is much more widely taught than Bayesian analyses, so NHST users are much more likely to be lowest common demoninator crank-turners who don’t really understand statistics generally. By contrast, if you’ve managed to find out how to do Bayesian inference, you’re probably better at statistics than the average researcher and therefore less likely to screw up whatever analysis you choose to do. If every researcher were taught Bayesian inference this would no longer be true.
Still, I think Bayesian methods are superior enough that the net benefit of that would be positive. (Also, proper Bayesian training would also cover how to construct ignorance priors, and I suspect nefariously chosen priors would be easier to spot than nefarious frequentist mistreatment of data.)
Bayesian methods are better in a number of ways, but ignorant people using a better tool won’t necessarily get better results. I don’t think the net effect of a mass switch to Bayesian methods would be negative, but I do think it’d be very small unless it involved raising the general statistical competence of scientists.
Even when Bayesian methods get so commonplace that they could be used just by pushing a button in SPSS, researchers will still have many tricks at their disposal to skew their conclusions. Not bothering to publish contrary data, only publishing subgroup analyses that show a desired result, ruling out inconvenient data points as “outliers”, wilful misinterpretation of past work, failing to correct for doing multiple statistical tests (and this can be an issue with Bayesian t-tests, like those in the Wagenmakers et al. reanalysis lukeprog linked above), and so on.
As a biologist, I can say that most statistical errors are just that: errors. They are not tricks. If researchers understand the statistics that they are using, a lot of these problems will go away.
A person has to learn a hell of a lot before they can do molecular biology research, and statistics happens to be fairly low on the priority list for most molecular biologists. In many situations we are able to get around the statistical complexities by generating data with very little noise.
Hanlon’s Razor FTW.
ISTM a large benefit of commonplace Bayes would be that competent statisticians could do actually meaningful meta-analyses...? Which would counteract widespread statistical ineptitude to a significant extent...?
I’m not sure it’d make much difference. From reading & skimming meta-analyses myself I’ve inferred that the main speedbumps with doing them are problems with raw data themselves or a lack of access to raw data. Whether the data were originally summarized using NHST/frequentist methods or Bayesian methods makes a lot less difference.
Edit to add: when I say “problems with raw data themselves” I don’t necessarily mean erroneous data; a problem can be as mundane as the sample/dataset not meeting the meta-analyst’s requirements (e.g. if the sample were unrepresentative, or the dataset didn’t contain a set of additional moderator variables).
I think that teaching Bayesian methods would itself raise the general statistical competence of scientists as a side effect, among other things because the meaning of p-values is seriously counter-intuitive (so more scientists would actually grok Bayesian statistics in such a world than actually grok frequentist statistics right now).
You could well be right. I’m pessimistic about this because I remember seeing lots of people at school & university recoiling from any statistical topic more advanced than calculating means and drawing histograms. If they were being taught about conjugate priors & hyperparameters I’d expect them to react as unenthusiastically as if they were being taught about confidence levels and maximum likelihood. But I don’t have any rock solid evidence for that hunch.