I’ve put a lot of thought into trying to improve the aggregation of science data in order to quickly determine the legitimacy of any particular paper based on current knowledge. Ultimately, it seems to me, aggregation isn’t viable because so much of the data is simply inaccessible (as you mentioned, accessibility is more a function of being new and exciting, whereas ‘vanilla’ building blocks such as replications may not be publicly accessible at all).
Ultimately, I think, the incentive structure has to change, and that’s difficult because when only positives are rewarded, it has the side effect of incentivizing the reporting of false positives, and disincentivizing putting in the work of reporting negatives (which are still useful!). Ultimately, I think the only viable ways to change the system are by convincing the grant providers to enforce certain guidelines. If they are as convinced as to the usefulness of non-positive and replication results as the rest of us, they can enforce reporting all results, and increase the rewards for people to do replications. Then once all that data is publicly available, you can do wonders with it.
I’d welcome other ideas, as like I said, I’ve been putting a lot of thought into it. I’d love to put together a system that lets people easily see the replication status and legitimacy of a paper (or scientific statement supported or opposed by various papers, which would be useful to the public at large), and I think I’ve puzzled out how, we just need to incentivize the people doing the research to release the data that will populate it.
I’ve put a lot of thought into trying to improve the aggregation of science data in order to quickly determine the legitimacy of any particular paper based on current knowledge. Ultimately, it seems to me, aggregation isn’t viable because so much of the data is simply inaccessible (as you mentioned, accessibility is more a function of being new and exciting, whereas ‘vanilla’ building blocks such as replications may not be publicly accessible at all).
Ultimately, I think, the incentive structure has to change, and that’s difficult because when only positives are rewarded, it has the side effect of incentivizing the reporting of false positives, and disincentivizing putting in the work of reporting negatives (which are still useful!). Ultimately, I think the only viable ways to change the system are by convincing the grant providers to enforce certain guidelines. If they are as convinced as to the usefulness of non-positive and replication results as the rest of us, they can enforce reporting all results, and increase the rewards for people to do replications. Then once all that data is publicly available, you can do wonders with it.
I’d welcome other ideas, as like I said, I’ve been putting a lot of thought into it. I’d love to put together a system that lets people easily see the replication status and legitimacy of a paper (or scientific statement supported or opposed by various papers, which would be useful to the public at large), and I think I’ve puzzled out how, we just need to incentivize the people doing the research to release the data that will populate it.