Satire of Journal of Personality and Social Psychology’s publication bias

[CarlShulman]

Follow-up to: Follow-up on ESP study: “We don’t publish replications”, Using degrees of freedom to change the past for fun and profit

As I discussed in the above posts, the Journal of Personality and Social Psychology, a leading psych journal, published a deeply flawed parapsychology study (see the second post for details) which had apparently been tortured to produce results. Then they rejected an attempt to replicate that found no effect, citing a sadly typical policy of not publishing replications. Some of you may enjoy reading one enterprising researcher’s amusing satire article, purportedly (not actually) “tallying” past confirmations and disconfirmations in JPSP and drawing conclusions.

ETA: To clarify the last sentence, they didn’t really find 4800+ confirmation and two disconfirmations. As they say in small print, the data were made up. It’s right by the chart.

[gwern]

The evidence for precognition is JPSP itself. Just open a random issue. In it, you’ll find countless a priori hypotheses anticipating the findings that eventually occurred. It is patently obvious that one could not possibly have anticipated the results without some form of precognition. Examples:

• People walk slower after thinking about moving to Florida?

• People are obsessed with thinking about white bears?

• People have, don’t have, have, don’t have, have personalities?

Skeptics may think that this evidence is just cherry-picking. Not so. Consider the sheer magnitude of hypothesis confirmation. The Figure presents the proportion of hypotheses by the authors of JPSP articles that were confirmed versus disconfirmed. All hypotheses except for two were confirmed. In one case, Denes-Raj and Epstein (1994) had a secondary hypothesis that their primary hypothesis would be incorrect. In the other case, Zajonc (1969), himself an alien with domination aspirations (Bones, 1996), declared that by 1974 cockroaches would control most of the eastern U.S. because “it would be so easy and they inspire each other so damn well.” Compared to the paltry accuracy rates by Bem’s precognition subjects (less than 60%!), the conclusion is clear: Bem’s effects are startlingly weak compared to the published evidence for precognition in JPSP.

Brilliant.

[Jack]

My favorite part:

Finally, a skeptic might counter that the JPSP authors could have conducted the studies, found results, dismissed inconsistent data, and then written the paper as if those were the results that they had anticipated all along. However, orchestrating such a large-scale hoax would require the coordination and involvement of thousands of researchers, reviewers, and editors. Researchers would have to selectively report those that “worked.” Reviewers and editors would have to selectively accept positive, confirmatory results and reject any norm violating researchers that submitted negative results. The possibility that an entire field could be perpetrating such a scam is so counterintuitive that only a social psychologist could predict it if it were actually true.

[Randaly]

This was actually published, in Perspectives on Psychological Science.

Some other “publications” by “Arina Bones” [actually Brian Nosek]:

Bones, A. K. & Gosling, S. D. (2009). Do Social Psychologists cause priming research or does priming research cause Social Psychologists? Poster presented at the Annual Meeting of the Society for Personality and Social Psychology, Tampa, FL. [Note: The 2nd author’s contribution was extracted from his memory under sedation. Any protests that he did not contribute should be ignored.]

Bones, A. K. & Johnson, N. R. (2007). Measuring the immeasurable, or, “Could Abraham Lincoln Take the Implicit Association Test?” Perspectives on Psychological Science, 2, 406-411.

Bones, A. K., & Johnson, N.R. (2002). Phil Zimbardo is definitely an alien. American Psychologist, 57, 1135–1142.

Bones, A. K. (1996). Invaders need no facilitators: Merely exposing the alien “Zajonc”. American Psychologist, 51, 1231–1238.

(Some of these are fake; some seem to have been actually published.)

[JoshuaZ]

The essential argument here is clever- their extremely high rate of turning up hypotheses that are confirmed suggest that they must have psychic abilities.

There’s a legitimate criticism here. But it may not be as legitimate as it seems at first glance. The most obvious problem here is publication bias (very bad), but there are other explanations as well, such as a tendency to write papers after the fact like the result was the initial hypothesis (bad but not nearly as bad as publication bias in terms of the long-term damage). Also, in areas like psychology, there’s another possibility- humans have some developed intuitions for psychological behavior. In that regard, we may do a better job intuiting the correct results in psychology than in other fields, and thus hypotheses are more likely to be correct in psychology. The result here is too extreme for this last bit to account for everything at hand, and certainly there are areas where human intuition about psychology is woefully bad, but this may account for a substantial fraction of what is going on here.

In any event, there really are serious problems in psychology as a subject as a whole, and the JPSP is not an isolated case.

[Will_Newsome]

In any event, there really are serious problems in psychology as a subject as a whole, and the JPSP is not an isolated case.

This is what I’ve been thinking quite often during Carl’s mini-series—beating up on people who believe in magical powers, or on a specific journal, is of course quite easy, but I’m worried that people won’t catch the more general and significantly more important point—e.g., that all those heuristics and biases results that are cited around here, might not be so trustworthy.

(ETA: Also, strange you wrote almost exactly the same comment I would have; I don’t think we normally have similar intuitions &c.)

[beoShaffer]

Somewhat related Einstein’s Arrogance.

[Jack]

This is amazing.