Question for clarification: I’m confused by what you mean when you say that the null hypothesis is always false. My understanding is that the null hypothesis is generally “these two sets of data were actually drawn from the same distribution”, and the hypothesis is generally “these two sets of data are drawn from different distributions, because the variable of interest which differs between them is in the casual path of a distributional change”.
Do you have a different way of viewing the null hypothesis in mind?
(I totally agree with your conclusion that likelihood ratios should be presented asking with p values, and also ideally other details about the data also.)
The null hypothesis in most papers is of the form “some real-valued random variable is equal to zero”. This could be an effect size, a regression coefficient, a vector of regression coefficients, et cetera.
If that’s your null hypothesis then the null hypothesis actually being true is an event of zero probability, in the sense that if your study had sufficient statistical power it would pick up on a tiny signal that would make the variable under consideration (statistically) significantly different from zero. If you believe there are no real-valued variables in the real world then it’s merely an event of probability ε>0 where ε is a tiny real number, and this distinction doesn’t matter for my purposes.
Incidentally, I think this is actually what happened with Bem’s parapsychology studies: his methodology was sound and he indeed picked up a small signal, but the signal was of experimenter effects and other problems in the experimental design rather than of paranormal phenomena. My claim is that no matter what you’re investigating, a sufficiently powerful study will always manage to pick up on such a small signal.
The point is that the null hypothesis being false doesn’t tell you much without a useful alternative hypothesis at hand. If someone tells you “populations 1 and 2 have different sample means”, the predictive value of that precise claim is nil.
Question for clarification: I’m confused by what you mean when you say that the null hypothesis is always false. My understanding is that the null hypothesis is generally “these two sets of data were actually drawn from the same distribution”, and the hypothesis is generally “these two sets of data are drawn from different distributions, because the variable of interest which differs between them is in the casual path of a distributional change”. Do you have a different way of viewing the null hypothesis in mind? (I totally agree with your conclusion that likelihood ratios should be presented asking with p values, and also ideally other details about the data also.)
The null hypothesis in most papers is of the form “some real-valued random variable is equal to zero”. This could be an effect size, a regression coefficient, a vector of regression coefficients, et cetera.
If that’s your null hypothesis then the null hypothesis actually being true is an event of zero probability, in the sense that if your study had sufficient statistical power it would pick up on a tiny signal that would make the variable under consideration (statistically) significantly different from zero. If you believe there are no real-valued variables in the real world then it’s merely an event of probability ε>0 where ε is a tiny real number, and this distinction doesn’t matter for my purposes.
Incidentally, I think this is actually what happened with Bem’s parapsychology studies: his methodology was sound and he indeed picked up a small signal, but the signal was of experimenter effects and other problems in the experimental design rather than of paranormal phenomena. My claim is that no matter what you’re investigating, a sufficiently powerful study will always manage to pick up on such a small signal.
The point is that the null hypothesis being false doesn’t tell you much without a useful alternative hypothesis at hand. If someone tells you “populations 1 and 2 have different sample means”, the predictive value of that precise claim is nil.
I understand much better now what you were saying. Thanks for clarifying.