How Stuart Buck funded the replication crisis

[Elizabeth]

Link post

Stuart Buck has perhaps the largest shapley value of any one individual in uncovering the replication crisis first in psychology, and then in many fields. This is his personal account of how and why he made the choices he did.

[jacquesthibs]

When you say “creating the replication crisis”, it read to me like he caused lots of people to publish papers that don’t replicate!

[Mo Putera]

A favorite essay of mine in the “personal anecdotes” department. (Stuart is also here on LW)

I’ll pull out some quotes I liked to entice folks to read the whole thing:

I.

From this point forward, I won’t narrate all of the grants and activities chronologically, but according to broader themes that are admittedly a bit retrofitted. Specifically, I’m now a fan of the pyramid of social change that Brian Nosek has written and talked about for a few years:

In other words, if you want scientists to change their behavior by sharing more data, you need to start at the bottom by making it possible to share data (i.e., by building data repositories). Then try to make it easier and more streamlined, so that sharing data isn’t a huge burden. And so on, up the pyramid.

You can’t start at the top of the pyramid (“make it required”) if the other components aren’t there first. For one thing, no one is going to vote for a journal or funder policy to mandate data sharing if it isn’t even possible. Getting buy-in for such a policy would require work to make data sharing not just possible, but more normative and rewarding within a field.

That said, I might add another layer at the bottom of the pyramid: “Raise awareness of the problem.” For example, doing meta-research on the extent of publication bias or the rate of replication can make entire fields aware that they have a problem in the first place—before that, they aren’t as interested in potential remedies for improving research behaviors.

The rest of this piece will be organized accordingly:

• Raise Awareness: fundamental research on the extent of irreproducibility;

• Make It Possible and Make It Easy: the development of software, databases, and other tools to help improve scientific practices;

• Make It Normative: journalists and websites that called out problematic research, and better standards/​guidelines/​ratings related to research quality and/​or transparency;

• Make It Rewarding: community-building efforts and new journal formats

• Make It Required: organizations that worked on policy and advocacy.

II.

On p-values and science communication done well:

In 2015, METRICS (the Meta-Research Innovation Center at Stanford) hosted an international conference on meta-research that was well-attended by many disciplinary leaders. The journalist Christie Aschwanden was there, and she went around ambushing the attendees (including me) by asking politely, “I’m a journalist, would you mind answering a few questions on video?,” and then following that with, “In layman’s terms, can you explain what is a p-value?” The result was a hilarious “educational” video and article, still available here. I was singled out as the one person with the “most straightforward explanation” of a p-value, but I did have an advantage — thanks to a job where I had to explain research issues on a daily basis to other foundation employees with little research background, I was already in the habit of boiling down complicated concepts.

(There’s a longer passage further down on Stuart’s experience consulting with the John Oliver Show where he rewrote the script on how to talk about p-values properly.)

III.

On how Stuart thinks his success as a “metascience venture capitalist” would’ve been far less if he’d been forced to project high EVs for each grant:

One grantee wrote to me:

“That grant was a real accelerator. The flexibility (which flows from trust, and confidence, in a funder) was critical in being able to grow, and do good work. It also helped set my expectations high around funders being facilitative rather than obstructive (possibly too high…). I think clueful funding is critical, I have seen funders hold projects and people back, not by whether they gave money, but how they gave it, and monitored work afterwards.”

To me, that captures the best of what philanthropy can do. Find great people, empower them with additional capital, and get out of their way.

By contrast, government funders make grants according to official criteria and procedures. Private philanthropy often acts the same way. As a result, there aren’t enough opportunities for innovative scientists or metascientists to get funding for their best ideas.

My own success as a metascience VC would have been far less if I had been forced to project a high expected-value for each grant. Indeed, such a requirement would have literally ruled out many of the highest-impact grants that I made (or else I would have been forced to produce bullshit projections).

The paradox is that the highest-impact work often cannot be predicted reliably in advance. Which isn’t that surprising. As in finance, the predictable activities that might lead to high impact are essentially priced into the market, because people and often entire organizations will already be working on those activities (often too much so!).

If you want to make additional impact beyond that, you’re left with activities that can’t be perfectly predicted and planned several years in advance, and that require some insight beyond what most peer reviewers would endorse.

What’s the solution? You have to rely on someone’s instinct or “nose” for smelling out ideas where the only articulable rationale is, “These people seem great and they’ll probably think of something good to do,” or “Not sure why, but this idea seems like it could be really promising.” In a way, it’s like riding a bicycle: it depends heavily on tacit and unarticulable knowledge, and if you tried to put everything in writing in advance, you would just make things worse.

Both public and private funders should look for more ways for talented program officers to hand out money (with few or no strings attached) to people and areas that they feel are promising. That sort of grantmaking might never be 100% when it comes to public funds at NIH or NSF, but maybe it could be 20%, just as a start. I suspect the results would be better than today, if only by increasing variance.