The essential argument here is clever- their extremely high rate of turning up hypotheses that are confirmed suggest that they must have psychic abilities.
There’s a legitimate criticism here. But it may not be as legitimate as it seems at first glance. The most obvious problem here is publication bias (very bad), but there are other explanations as well, such as a tendency to write papers after the fact like the result was the initial hypothesis (bad but not nearly as bad as publication bias in terms of the long-term damage). Also, in areas like psychology, there’s another possibility- humans have some developed intuitions for psychological behavior. In that regard, we may do a better job intuiting the correct results in psychology than in other fields, and thus hypotheses are more likely to be correct in psychology. The result here is too extreme for this last bit to account for everything at hand, and certainly there are areas where human intuition about psychology is woefully bad, but this may account for a substantial fraction of what is going on here.
In any event, there really are serious problems in psychology as a subject as a whole, and the JPSP is not an isolated case.
In any event, there really are serious problems in psychology as a subject as a whole, and the JPSP is not an isolated case.
This is what I’ve been thinking quite often during Carl’s mini-series—beating up on people who believe in magical powers, or on a specific journal, is of course quite easy, but I’m worried that people won’t catch the more general and significantly more important point—e.g., that all those heuristics and biases results that are cited around here, might not be so trustworthy.
(ETA: Also, strange you wrote almost exactly the same comment I would have; I don’t think we normally have similar intuitions &c.)
The essential argument here is clever- their extremely high rate of turning up hypotheses that are confirmed suggest that they must have psychic abilities.
There’s a legitimate criticism here. But it may not be as legitimate as it seems at first glance. The most obvious problem here is publication bias (very bad), but there are other explanations as well, such as a tendency to write papers after the fact like the result was the initial hypothesis (bad but not nearly as bad as publication bias in terms of the long-term damage). Also, in areas like psychology, there’s another possibility- humans have some developed intuitions for psychological behavior. In that regard, we may do a better job intuiting the correct results in psychology than in other fields, and thus hypotheses are more likely to be correct in psychology. The result here is too extreme for this last bit to account for everything at hand, and certainly there are areas where human intuition about psychology is woefully bad, but this may account for a substantial fraction of what is going on here.
In any event, there really are serious problems in psychology as a subject as a whole, and the JPSP is not an isolated case.
This is what I’ve been thinking quite often during Carl’s mini-series—beating up on people who believe in magical powers, or on a specific journal, is of course quite easy, but I’m worried that people won’t catch the more general and significantly more important point—e.g., that all those heuristics and biases results that are cited around here, might not be so trustworthy.
(ETA: Also, strange you wrote almost exactly the same comment I would have; I don’t think we normally have similar intuitions &c.)
Somewhat related Einstein’s Arrogance.