(Caveat: I’m not an expert in this field) -- I expect there could be some value in a ~‘registered reports’ approach for these high-cost computational experiments.
In informal reporting (ACX, this forum) recall reading some mentions of something related to the “publication bias” story in econ/social sci. Perhaps more like concerns of ‘labs reporting selectively’; both researchers promoting capabilities (selective reporting on successes) and safety-aligned researchers accused of cherry-picking the most alarming failures/misalignment evidence.
Yea, I can definitely see the selective reporting problem, which goes beyond the problem of negative results being unfairly denied publication. But to combat selective reporting, you’d really need to require preregistered experiments, which is more of a collective-action problem between journals, since if any of them allow un-preregistered experiments, the authors can just publish there. (Of course, you can try and convince the broad community to ignore all experiments that aren’t preregistered, but if you can do this then you’ve already won; the journals will be strongly incentivized to follow suit.)
Required preregistration is just very cumbersome and difficult to do for exploratory science; it really seems only feasible for the later stages of things like medical trials or big contentious question requiring a decisive experiment.
(Caveat: I’m not an expert in this field) -- I expect there could be some value in a ~‘registered reports’ approach for these high-cost computational experiments.
In informal reporting (ACX, this forum) recall reading some mentions of something related to the “publication bias” story in econ/social sci. Perhaps more like concerns of ‘labs reporting selectively’; both researchers promoting capabilities (selective reporting on successes) and safety-aligned researchers accused of cherry-picking the most alarming failures/misalignment evidence.
Yea, I can definitely see the selective reporting problem, which goes beyond the problem of negative results being unfairly denied publication. But to combat selective reporting, you’d really need to require preregistered experiments, which is more of a collective-action problem between journals, since if any of them allow un-preregistered experiments, the authors can just publish there. (Of course, you can try and convince the broad community to ignore all experiments that aren’t preregistered, but if you can do this then you’ve already won; the journals will be strongly incentivized to follow suit.)
Required preregistration is just very cumbersome and difficult to do for exploratory science; it really seems only feasible for the later stages of things like medical trials or big contentious question requiring a decisive experiment.