Now, if that is a fair summary, then this big controversy between frequentists and Bayesians must mean that there is a sizable collection of people who think that the above procedure is a better way of obtaining knowledge than performing Bayesian updates.
Not necessarily better. Just more convenient for the thumbs up/thumbs down way of looking at evidence that scientists tend to like.
But for the life of me, I can’t see how anyone could possibly think that. I mean, not only is the “p-value” threshold arbitrary,
It’s a convention. The point is to have a pre-agreed, low significance level so that testers can’t screw with the result of a test by arbitrary jacking the significance level up (if they want to reject a hypothesis) or turning it down (if they don’t). The significance level has to be low to minimize the risk of a type I error.
not only are we depriving ourselves of valuable information by “accepting” or “not accepting” a hypothesis rather than quantifying our certainty level,
The certainty level is effectively communicated via the significance level and p-value itself. (And the use of a reject vs. don’t reject dichotomy can be desirable if one wishes to decide between performing some action and not performing it based on some data.)
but...what about P(E|H)?? (Not to mention P(H).) To me, it seems blatantly obvious that an epistemology (and that’s what it is) like the above is a recipe for disaster—specifically in the form of accumulated errors over time.
A frequentist can deal in likelihoods, for example by doing hypothesis tests of likelihood ratios. As for priors, a frequentist encapsulates them in parametric and sampling assumptions about the data. A Bayesian might give a low weight to a positive result from a parapsychology study because of their “low priors”, but a frequentist might complain about sampling procedures or cherrypicking being more likely than a true positive. As I see it, the two say essentially the same thing; the frequentist is just being more specific than the Bayesian.
I’m not seeing why what you call “the real WTF” is evidence of a problem with frequentist statistics. The fact that the hypothesis test would have given a statistically insignificant p-value whatever the actual 6 data points were just indicates that whatever the population distributions, 6 data points are simply not enough to disconfirm the null hypothesis. In fact you can see this if you look at Mann & Whitney’s original paper! (See the n=3 subtable in table I, p. 52.)
I can picture someone counterarguing that this is not immediately obvious from the details of the statistical test, but I would hope that any competent statistician, frequentist or not, would be sceptical of a nonparametric comparison of means for samples of size 3!