I found that so-called parapsychology research is suffering from the p-value problem badly. In one book I read some thing like that they tossed a coin 1000 times, and it came 520 heads. The probability of 520 heads from 1000 is like p = 0.01, so they concluded that their result is significant. However, it is still inside standard deviation from 500.
This helped me better understand the problem with p-values: even if they got 500 head and 500 tails, it would still have p-value around 1 per cent. But if any psi-effect were true, their result should be outside standard deviation. In other words, whet they should measure was not the probability of the result, but such probability of the result given their hypothesis.
What does “inside standard deviation from 500” mean?
Having a small p-value is exactly the same thing, at least for approximately normally distributed things like this, as being multiple standard deviations away from the norm.
The specific number here is neither “like p=0.01” nor within 1σ of the mean. Variance of a binomial distribution is npq=250, so standard deviation is just under 16. Being at least 20 away from 500 is approximately a p=0.2 event.
I found that so-called parapsychology research is suffering from the p-value problem badly. In one book I read some thing like that they tossed a coin 1000 times, and it came 520 heads. The probability of 520 heads from 1000 is like p = 0.01, so they concluded that their result is significant. However, it is still inside standard deviation from 500.
This helped me better understand the problem with p-values: even if they got 500 head and 500 tails, it would still have p-value around 1 per cent. But if any psi-effect were true, their result should be outside standard deviation. In other words, whet they should measure was not the probability of the result, but such probability of the result given their hypothesis.
What does “inside standard deviation from 500” mean?
Having a small p-value is exactly the same thing, at least for approximately normally distributed things like this, as being multiple standard deviations away from the norm.
The specific number here is neither “like p=0.01” nor within 1σ of the mean. Variance of a binomial distribution is npq=250, so standard deviation is just under 16. Being at least 20 away from 500 is approximately a p=0.2 event.