I had definitely noticed all the different nutrition, psychology, and biological initiatives like OSF or the Reproducibility Project, and how expensive they all are, but I didn’t realize that they all owed their funding to a single source. (I had only ever briefly heard of Arnold in the context of pension reform.) I’m very glad Arnold is doing this, but I now feel more pessimistic about academia than when I assumed that the funding for all this was coming from a broad coalition of universities and nonprofits etc....
When evaluating whether there is a broad base of support, I think it’s important to distinguish “one large-scale funder” from “narrow overall base of support”. Before the Arnold foundation’s funding, the reproducibility project had a broad base of committed participants contributing their personal resources and volunteering their time.
To add some details from personal experience: In late 2011 and early 2012, the Reproducibility Project was a great big underfunded labor of love. Brian Nosek had outlined a plan to replicate ~50 studies—this became the Science 2015 paper. He was putting together spreadsheets to coordinate everyone, and hundreds of researchers who were personally committed to the cause were allocating their own discretionary funds and working in their spare time to get the replications done in their own labs. The mailing list was thriving. Researchers were paying subjects out-of-pocket. Reproducibility wasn’t a full-blown memetic explosion in the public eye, nor was there a major source of funding, but we were getting notable media coverage, and researchers kept joining.
Importantly, I think we were already firmly on track to write the 2015 Science paper before the Arnold Foundation took notice of the coverage that existing projects were getting and began reaching out to Nosek and others to ask if they could do more with more funding.
When the Center for Open Science was founded, it increased the scale of coordination that Brian and other coordinators were able to execute amongst participants. I’d guess that Brian himself was also able to spend more time talking to the media. The base of participating researchers remained broad and unpaid. I’d guess that the vast majority of researchers contributing personally to the reproducibility movement are still not getting any earmarked funds for it.
I wasn’t aware of the details of COS’s funding before reading this article, so I have no additional evidence about whether there are more large-scale funders. A brief round of Googling turns up a few other Open Science flavored sources of money (e.g. https://www.openscienceprize.org/res/p/FAQ/) but these are not specific to reproducibility; rather, they’re more broadly targeted towards open sharing of code, data, and methods.
A few suggested takeaways:
There may be other cases where an existing movement with an enthusiastic base of participants is funding-limited in the scale of coordination and publicity they can achieve, and a single motivated funder can make a substantial impact by adding that type of funding.
Under “normal” funding and incentive conditions, the reproducibility project was able to form and begin producing concrete and impactful output, but thereafter it appears that only one major source of funding materialized and no other dedicated large-scale funding has been available. I think this should make you feel optimistic about academic researchers as individuals and as a culture, but pessimistic about traditional academic funding routes, rather than monolithically pessimistic about academia as a whole.
Seems like its more that the institutions are broken rather than few people caring. Or could be that most scientists don’t care that much but a significant minority care a lot. And for that to cause lots of change you need money, but to get money you need the traditional funders (who don’t care because most scientists don’t care) or you need outside help.
The underlying incentives for modern scientists are really skewed away from truth seeking. The modern publish or perish paradigm embodied by things like RAE and REF encourage flashiness and being “world-leading”. I’m guessing the rigour metric (being the least quantifiable?) is the least optimised for.
It seems he is trying to culture shift the wrong people. It is not scientists that provide the publish or perish incentives it is the schools and the bureaucrats.
It would be interesting to figure out what kind of bureaucratic incentive system would encourage truth seeking for academics.
″ Laura and John Arnold didn’t start the movement to reform science, but they have done more than anyone else to amplify its capabilities—typically by approaching researchers out of the blue and asking whether they might be able to do more with more money. “The Arnold Foundation has been the Medici of meta-research,” Ioannidis says. All told, the foundation’s Research Integrity initiative has given more than $80 million to science critics and reformers in the past five years alone.”
I had definitely noticed all the different nutrition, psychology, and biological initiatives like OSF or the Reproducibility Project, and how expensive they all are, but I didn’t realize that they all owed their funding to a single source. (I had only ever briefly heard of Arnold in the context of pension reform.) I’m very glad Arnold is doing this, but I now feel more pessimistic about academia than when I assumed that the funding for all this was coming from a broad coalition of universities and nonprofits etc....
When evaluating whether there is a broad base of support, I think it’s important to distinguish “one large-scale funder” from “narrow overall base of support”. Before the Arnold foundation’s funding, the reproducibility project had a broad base of committed participants contributing their personal resources and volunteering their time.
To add some details from personal experience: In late 2011 and early 2012, the Reproducibility Project was a great big underfunded labor of love. Brian Nosek had outlined a plan to replicate ~50 studies—this became the Science 2015 paper. He was putting together spreadsheets to coordinate everyone, and hundreds of researchers who were personally committed to the cause were allocating their own discretionary funds and working in their spare time to get the replications done in their own labs. The mailing list was thriving. Researchers were paying subjects out-of-pocket. Reproducibility wasn’t a full-blown memetic explosion in the public eye, nor was there a major source of funding, but we were getting notable media coverage, and researchers kept joining.
Importantly, I think we were already firmly on track to write the 2015 Science paper before the Arnold Foundation took notice of the coverage that existing projects were getting and began reaching out to Nosek and others to ask if they could do more with more funding.
When the Center for Open Science was founded, it increased the scale of coordination that Brian and other coordinators were able to execute amongst participants. I’d guess that Brian himself was also able to spend more time talking to the media. The base of participating researchers remained broad and unpaid. I’d guess that the vast majority of researchers contributing personally to the reproducibility movement are still not getting any earmarked funds for it.
I wasn’t aware of the details of COS’s funding before reading this article, so I have no additional evidence about whether there are more large-scale funders. A brief round of Googling turns up a few other Open Science flavored sources of money (e.g. https://www.openscienceprize.org/res/p/FAQ/) but these are not specific to reproducibility; rather, they’re more broadly targeted towards open sharing of code, data, and methods.
A few suggested takeaways:
There may be other cases where an existing movement with an enthusiastic base of participants is funding-limited in the scale of coordination and publicity they can achieve, and a single motivated funder can make a substantial impact by adding that type of funding.
Under “normal” funding and incentive conditions, the reproducibility project was able to form and begin producing concrete and impactful output, but thereafter it appears that only one major source of funding materialized and no other dedicated large-scale funding has been available. I think this should make you feel optimistic about academic researchers as individuals and as a culture, but pessimistic about traditional academic funding routes, rather than monolithically pessimistic about academia as a whole.
Thanks, this was super useful context.
Seems like its more that the institutions are broken rather than few people caring. Or could be that most scientists don’t care that much but a significant minority care a lot. And for that to cause lots of change you need money, but to get money you need the traditional funders (who don’t care because most scientists don’t care) or you need outside help.
The underlying incentives for modern scientists are really skewed away from truth seeking. The modern publish or perish paradigm embodied by things like RAE and REF encourage flashiness and being “world-leading”. I’m guessing the rigour metric (being the least quantifiable?) is the least optimised for.
It seems he is trying to culture shift the wrong people. It is not scientists that provide the publish or perish incentives it is the schools and the bureaucrats.
It would be interesting to figure out what kind of bureaucratic incentive system would encourage truth seeking for academics.
Good article in Wired
″ Laura and John Arnold didn’t start the movement to reform science, but they have done more than anyone else to amplify its capabilities—typically by approaching researchers out of the blue and asking whether they might be able to do more with more money. “The Arnold Foundation has been the Medici of meta-research,” Ioannidis says. All told, the foundation’s Research Integrity initiative has given more than $80 million to science critics and reformers in the past five years alone.”