The details are hazy at this point, but by assigning a realistic probability to the “Something else” hypothesis, you avoid making over confident estimates of your other hypotheses in a multiple hypothesis testing scenario.
See Multiple Hypothesis Testing in Jaynes PTTLOS, starting pg. 98, and the punchline on pg. 105:
In summary, the role of our new hypothesis C was only to be held in abeyace until needed, like a fire extinguisher. In a normal testing situation, it is “dead”, playing no part in the inference because its probability remains far below that of the other hypotheses. But a dead hypothesis can be brought back to life by very unexpected data.
I think this is especially relevant to standard “null hypothesis” hypothesis testing because the likelihood of the data under the alternative hypothesis is never calculated, so you don’t even get a hint that your model might just suck, and instead conclude that the null hypothesis should be rejected.
What is the likelihood of the “something else” hypothesis? I don’t think this is really a general remedy.
Also, you can get the same thing in the hypothesis testing framework by doing two hypothesis tests, one of which is a comparison to the “something else” hypothesis and one of which is a comparison to the original null hypothesis.
Finally, while I forgot to mention this above, in most cases where hypothesis testing is applied, you actually are considering all possibilities, because you are doing something like P0 = “X ⇐ 0”, P1 = “X > 0″ and these really are logically the only possibilities =) [although I guess often you need to make some assumptions on the probabilistic dependencies among your samples to get good bounds].
Yes, you can say it in that framework. And you should. That’s part of the steelmanning exercise—putting in the things that are missing. If you steelman enough, you get to be a good bayesian.
P0 = “X ⇐ 0” and {All My other assumptions} NOT(P0) = NOT(“X ⇐ 0″) or NOT({All My other assumptions})
The details are hazy at this point, but by assigning a realistic probability to the “Something else” hypothesis, you avoid making over confident estimates of your other hypotheses in a multiple hypothesis testing scenario.
See Multiple Hypothesis Testing in Jaynes PTTLOS, starting pg. 98, and the punchline on pg. 105:
I think this is especially relevant to standard “null hypothesis” hypothesis testing because the likelihood of the data under the alternative hypothesis is never calculated, so you don’t even get a hint that your model might just suck, and instead conclude that the null hypothesis should be rejected.
What is the likelihood of the “something else” hypothesis? I don’t think this is really a general remedy.
Also, you can get the same thing in the hypothesis testing framework by doing two hypothesis tests, one of which is a comparison to the “something else” hypothesis and one of which is a comparison to the original null hypothesis.
Finally, while I forgot to mention this above, in most cases where hypothesis testing is applied, you actually are considering all possibilities, because you are doing something like P0 = “X ⇐ 0”, P1 = “X > 0″ and these really are logically the only possibilities =) [although I guess often you need to make some assumptions on the probabilistic dependencies among your samples to get good bounds].
Yes, you can say it in that framework. And you should. That’s part of the steelmanning exercise—putting in the things that are missing. If you steelman enough, you get to be a good bayesian.
P0 = “X ⇐ 0” and {All My other assumptions}
NOT(P0) = NOT(“X ⇐ 0″) or NOT({All My other assumptions})