A brilliant post with many links to the Yudkowsky Canon. It has just become a bookmark.
One quip: the study which revealed that a majority of research findings were false seemed to rely on a simulation, and on one meta-study performed earlier by the group. Have I understood this correctly?
Perhaps the p for biological experiments should be lower, but my first inclination is to defend the field I work in and its custom of p<0.05 .
Every time I open up an animal for surgery, the animals nerves are lying in slightly different places. There is a different amount of muscle-tissue obscuring the nerve I have to record from. I have to re-manufacture my electrode after every few surgeries. The hook on my electrode is differently shaped each time. When I put the nerve on the electrode, I pull it up with a different amount of force, in a different position, and do a different degree of damage to the nerve’s processes.
Every animal is slightly different, and every surgery goes slightly differently.
Therefore, I think it might be understandable, why I need a p that is higher than a physicists, because if the physicist uses the same stock of materials, he has much less variability in his setup to begin with.
That sounds to me more like an argument for needing lower p-values, not higher ones. If there are many confounding factors, you need a higher threshold of evidence for claiming that you are seeing a real effect.
Physicists need low p-values for a different reason, namely that they do very large numbers of statistical tests. If you choose p=0.05 as your threshold then it means that you are going to be claiming a false detection at least one time in twenty (roughly speaking), so if physicists did this they would be claiming false detections every other day and their credibility would plummet like a rock.
A brilliant post with many links to the Yudkowsky Canon. It has just become a bookmark.
One quip: the study which revealed that a majority of research findings were false seemed to rely on a simulation, and on one meta-study performed earlier by the group. Have I understood this correctly?
Perhaps the p for biological experiments should be lower, but my first inclination is to defend the field I work in and its custom of p<0.05 .
Every time I open up an animal for surgery, the animals nerves are lying in slightly different places. There is a different amount of muscle-tissue obscuring the nerve I have to record from. I have to re-manufacture my electrode after every few surgeries. The hook on my electrode is differently shaped each time. When I put the nerve on the electrode, I pull it up with a different amount of force, in a different position, and do a different degree of damage to the nerve’s processes.
Every animal is slightly different, and every surgery goes slightly differently.
Therefore, I think it might be understandable, why I need a p that is higher than a physicists, because if the physicist uses the same stock of materials, he has much less variability in his setup to begin with.
That sounds to me more like an argument for needing lower p-values, not higher ones. If there are many confounding factors, you need a higher threshold of evidence for claiming that you are seeing a real effect.
Physicists need low p-values for a different reason, namely that they do very large numbers of statistical tests. If you choose p=0.05 as your threshold then it means that you are going to be claiming a false detection at least one time in twenty (roughly speaking), so if physicists did this they would be claiming false detections every other day and their credibility would plummet like a rock.