“The Journal of Real Effects”
Luke’s recent post mentioned that The Lancet has a policy encouraging the advance registration of clinical trials, while mine examined an apparent case study of data-peeking and on-the-fly transformation of studies. But how much variation is there across journals on such dimensions? Are there journals that buck the standards of their fields (demanding registration, p=0.01 rather than p=0.05 where the latter is typical in the field, advance specification of statistical analyses and subject numbers, etc)? What are some of the standouts? Are there fields without any such?
I wonder if there is a niche for a new open-access journal, along the lines of PLoS, with standards strict enough to reliably exclude false-positives. Some possible titles:
The Journal of Real Effects
(Settled) Science
Probably True
Journal of Non-Null Results, Really
Too Good to Be False
_________________?
“Journal” might be a wrong concept, this service looks like something that could be decoupled from publishing.
I imagine it would not contain articles, only information about reliability of statements. Such as “result R published in article A was replicated successfully, here are the relevant data”. So when someone reads an article, they could come to this service to check its reliability.
How should such thing be organized?
Wikis, Consumer Reports, Cochrane Reports, etc. The virtue of journals would be to reduce switching costs for academia.
I know there are already journals which require pre-registration; perhaps one could investigate how successful they have been. (We’ve known about these issues for a long time, but they haven’t been fixed—which suggests incentive problems....)
Yes, I’m curious about the barriers if anyone knows more.
Would one get banned from this journal for registering the testing of 100 only slightly different hypotheses at p=0.01?