Probability theory still applies.
Ah, but which probability theory? Bayesian or frequentist? Or the ideas of Fisher?
How do you feel about the likelihood principle? The Behrens-Fisher problem, particularly when the variances are unknown and not assumed to be equal? The test of a sharp (or point) null hypothesis?
It does no good to assume that one’s statistics and probability theory are not built on axioms themselves. I have rarely met a probabilist or statistician whose answer about whether he or she believes in the likelihood principle or in the logically contradicted significance tests (or in various solutions of the Behrens-Fisher problem) does not depend on some sort of axiom or idea of what simply “seems right.” Of course, there are plenty of scientists who use mutually contradictory statistical tests, depending on what they’re doing.
A calculated probability of 0.0000001 should diminish the emotional strength of any anticipation, positive or negative, by a factor of ten million.
And there goes Walter Mitty and Calvin, then. If it is justifiable to enjoy art or sport, why is it not justifiable to enjoy gambling for its own sake?
if the results are significant at the 0.05 confidence level. Now this is not just a ritualized tradition. This is not a point of arbitrary etiquette like using the correct fork for salad.
The use of the 0.05 confidence level is itself a point of arbitrary etiquette. The idea that results close to identical, yet one barely meeting the arbitrary 0.05 confidence level and the other not, can be separated into two categories of “significant” and “not significant” is a ritualized tradition indeed perhaps not understood by many scientists. There are important reasons for having an arbitrary point to mark significance, and of having that custom be the same throughout science (and not chosen by the experimenter). But the actual point is arbitrary etiquette.
The commonality of utensils or traffic signals in a culture is important, even though the specific forms that they take are arbitrary. The exact confidence level used is arbitrary; it’s important that there is a standard.
Nor is Bayes’s Theorem different from one place to another.
No, but the statistical concept of “confidence” depends on how an experimenter thinks that a study was designed. See for example this discussion of the likelihood principle.
If Alice conducts 12 trials with 3 successes and 9 failures, do we reject the null hypothesis p = .5 versus p < .5 at the 0.05 confidence level? It turns out that the answer depends in the classical frequentist sense on whether Alice decided ahead of time to conduct 12 trials or decided to conduct trials until 3 successes were achieved. What if Alice drops dead after recording the results of the trials but not the setup? Then Bob and Chuck, finding the notebook, may disagree about significance. The “significance” depends on the design of the experiment rather than the results alone, according to classical methods.
How many scientists understand that?