I disagree with most of the post and most of the comments here. I think most academics are not explicitly committing fraud, but bad science results anyway. I also think that for the vast majority of (non-tenured) academics, if you don’t follow the incentives, you don’t make it in academia. If you intervened on ~100 entering PhD students and made them committed to always not following the incentives where they are bad, I predict that < 10% of them will become professors—maybe an expected 2 of them would. So you can’t say “why don’t the academics just not follow the incentives”; any such person wouldn’t have made it into academia. I think the appropriate worlds to consider are: science as it exists now with academics following incentives or ~no academia at all.
It is probably correct that each individual instance of having to deal with bad incentives doesn’t make that much of a difference, but there are many such instances. Probably there’s an 80-20 thing to do here where you get 80% of the benefit by not following the worst 20% of bad incentives, but it’s actually quite hard to identify these, and it requires you to be able to predict the consequences of not following the bad incentives, which is really hard to do. (I don’t think I could do it, and I’ve been in a PhD program for 5 years now.)
To be clear: if you know that someone explicitly and intentionally committed fraud for personal gain with the knowledge that it would result in bad science, that seems fine to punish. But this is rare, and it’s easy to mistake well-intentioned mistakes for intentional fraud.
Survey and other data indicate that in these fields most people were doing p-hacking/QRPs (running tests selected ex post, optional stopping, reporting and publication bias, etc), but a substantial minority weren’t, with individual, subfield, and field variation. Some people produced ~100% bogus work while others were ~0%. So it was possible to have a career without the bad practices Yarkoni criticizes, aggregating across many practices to look at overall reproducibility of research.
And he is now talking about people who have been informed about the severe effects of the QRPs (that they result in largely bogus research at large cost to science compared to reproducible alternatives that many of their colleagues are now using and working to reward) but choose to continue the bad practices. That group is also disproportionately tenured, so it’s not a question of not getting a place in academia now, but of giving up on false claims they built their reputation around and reduced grants and speaking fees.
I think the core issue is that even though the QRPs that lead to mostly bogus research in fields such as social psych and neuroimaging often started off without intentional bad conduct, their bad effects have now become public knowledge, and Yarkoni is right to call out those people on continuing them and defending continuing them.
Some people produced ~100% bogus work while others were ~0%. So it was possible to have a career without the bad practices Yarkoni criticizes, aggregating across many practices to look at overall reproducibility of research.
I’m curious how many were able to hit 0%? Based on my 10x estimate below I’d estimate 9%, but that was definitely a number I pulled out of nowhere.
That group is also disproportionately tenured, so it’s not a question of not getting a place in academia now, but of giving up on false claims they built their reputation around and reduced grants and speaking fees.
I personally feel the most pressure to publish because the undergrads I work with need a paper to get into grad school. I wonder if it’s similar for tenured professors with their grad students.
Also, the article seems to be condemning academics who are not tenured, e.g.
“I would publish in open access journals,” your friendly neighborhood scientist will say. “But those have a lower impact factor, and I’m up for tenure in three years.”
I think the core issue is that even though the QRPs that lead to mostly bogus research in fields such as social psych and neuroimaging often started off without intentional bad conduct, their bad effects have now become public knowledge, and Yarkoni is right to call out those people on continuing them and defending continuing them.
Thought experiment (that I acknowledge is not reality): Suppose that it were actually the case that in order to stay in academia you had to engage in QRPs. Do you still think it is right to call out / punish such people? It seems like this ends up with you always punishing everyone in academia, with no gains to actually published research, or you abolish academia outright.
The former is a statement about outcomes while the latter is a statement about intentions.
My model for how most academics end up following bad incentives is that they pick up the incentivized bad behaviors via imitation. Anyone who doesn’t do this ends up doing poorly and won’t make it in academia (and in any case such people are rare, imitation is the norm for humans in general). As part of imitation, people come up with explanations for why the behavior is necessary and good for them to do. (And this is also usually the right thing to do; if you are imitating a good behavior, it makes sense to figure out why it is good, so that you can use that underlying explanation to reason about what other behaviors are good.)
I think that I personally am engaging in bad behaviors because I incorrectly expect that they are necessary for some goal (e.g. publishing papers to build academic credibility). I just can’t tell which ones really are necessary and which ones aren’t.
Agreed that it’s related, and I do think it’s part of the explanation.
I will go even further: while in that post the selection happens at the level of properties of individuals who participate in some culture, I’m claiming that the selection happens at the higher level of norms of behavior in the culture, because most people are imitating the rest of the culture.
This requires even fewer misaligned individuals. Under the model where you select on individuals, you would still need a fairly large number of people to have the property of interest—if only 1% of salesmen had the personality traits leading to them being scammy and the other 99% were usually honest about the product, the scammy salesmen probably wouldn’t be able to capture all of the sales jobs. However, if most people imitate, then those 1% of salesmen will slowly push the norms towards being more scammy over generations, and you’d end up in the equilibrium where nearly every salesman is scammy.
Come to think of it, I think I would estimate that ~1% of academics are explicitly thinking about how to further their own career at the cost of science (in ways that are different from imitation).
If you intervened on ~100 entering PhD students and made them committed to always not following the incentives where they are bad, I predict that < 10% of them will become professors—maybe an expected 2 of them would.
And how many if you didn’t intervene?
So you can’t say “why don’t the academics just not follow the incentives”; any such person wouldn’t have made it into academia.
How do you reconcile this with the immediately prior sentence?
Significantly more, maybe 20. To do a proper estimate I’d need to know which field we’re considering, what the base rates are, etc. The thing I should have said was that I expect it makes it ~10x less likely that you become a professor; that seems more robust to the choice of field and isn’t conditional on base rates that I don’t know.
The Internet suggests a base rate of 3-5%, which means without intervention 3-5 of them would become professors; if that’s true I would say that with intervention an expected 0.4 of them would become professors.
How do you reconcile this with the immediately prior sentence?
I didn’t mean that it was literally impossible for a person who doesn’t follow the incentives to get into academia, I meant that it was much less likely. I do in fact know people in academia who I think are reasonably good at not following bad incentives.
I disagree with most of the post and most of the comments here. I think most academics are not explicitly committing fraud, but bad science results anyway. I also think that for the vast majority of (non-tenured) academics, if you don’t follow the incentives, you don’t make it in academia. If you intervened on ~100 entering PhD students and made them committed to always not following the incentives where they are bad, I predict that < 10% of them will become professors—maybe an expected 2 of them would. So you can’t say “why don’t the academics just not follow the incentives”; any such person wouldn’t have made it into academia. I think the appropriate worlds to consider are: science as it exists now with academics following incentives or ~no academia at all.
It is probably correct that each individual instance of having to deal with bad incentives doesn’t make that much of a difference, but there are many such instances. Probably there’s an 80-20 thing to do here where you get 80% of the benefit by not following the worst 20% of bad incentives, but it’s actually quite hard to identify these, and it requires you to be able to predict the consequences of not following the bad incentives, which is really hard to do. (I don’t think I could do it, and I’ve been in a PhD program for 5 years now.)
To be clear: if you know that someone explicitly and intentionally committed fraud for personal gain with the knowledge that it would result in bad science, that seems fine to punish. But this is rare, and it’s easy to mistake well-intentioned mistakes for intentional fraud.
Survey and other data indicate that in these fields most people were doing p-hacking/QRPs (running tests selected ex post, optional stopping, reporting and publication bias, etc), but a substantial minority weren’t, with individual, subfield, and field variation. Some people produced ~100% bogus work while others were ~0%. So it was possible to have a career without the bad practices Yarkoni criticizes, aggregating across many practices to look at overall reproducibility of research.
And he is now talking about people who have been informed about the severe effects of the QRPs (that they result in largely bogus research at large cost to science compared to reproducible alternatives that many of their colleagues are now using and working to reward) but choose to continue the bad practices. That group is also disproportionately tenured, so it’s not a question of not getting a place in academia now, but of giving up on false claims they built their reputation around and reduced grants and speaking fees.
I think the core issue is that even though the QRPs that lead to mostly bogus research in fields such as social psych and neuroimaging often started off without intentional bad conduct, their bad effects have now become public knowledge, and Yarkoni is right to call out those people on continuing them and defending continuing them.
I’m curious how many were able to hit 0%? Based on my 10x estimate below I’d estimate 9%, but that was definitely a number I pulled out of nowhere.
I personally feel the most pressure to publish because the undergrads I work with need a paper to get into grad school. I wonder if it’s similar for tenured professors with their grad students.
Also, the article seems to be condemning academics who are not tenured, e.g.
Thought experiment (that I acknowledge is not reality): Suppose that it were actually the case that in order to stay in academia you had to engage in QRPs. Do you still think it is right to call out / punish such people? It seems like this ends up with you always punishing everyone in academia, with no gains to actually published research, or you abolish academia outright.
Isn’t “academics who don’t follow bad incentives almost never become professors” blatantly incompatible with “these are well-intentioned mistakes”?
The former is a statement about outcomes while the latter is a statement about intentions.
My model for how most academics end up following bad incentives is that they pick up the incentivized bad behaviors via imitation. Anyone who doesn’t do this ends up doing poorly and won’t make it in academia (and in any case such people are rare, imitation is the norm for humans in general). As part of imitation, people come up with explanations for why the behavior is necessary and good for them to do. (And this is also usually the right thing to do; if you are imitating a good behavior, it makes sense to figure out why it is good, so that you can use that underlying explanation to reason about what other behaviors are good.)
I think that I personally am engaging in bad behaviors because I incorrectly expect that they are necessary for some goal (e.g. publishing papers to build academic credibility). I just can’t tell which ones really are necessary and which ones aren’t.
This seems related to the ideas in this post on unconscious economies.
Agreed that it’s related, and I do think it’s part of the explanation.
I will go even further: while in that post the selection happens at the level of properties of individuals who participate in some culture, I’m claiming that the selection happens at the higher level of norms of behavior in the culture, because most people are imitating the rest of the culture.
This requires even fewer misaligned individuals. Under the model where you select on individuals, you would still need a fairly large number of people to have the property of interest—if only 1% of salesmen had the personality traits leading to them being scammy and the other 99% were usually honest about the product, the scammy salesmen probably wouldn’t be able to capture all of the sales jobs. However, if most people imitate, then those 1% of salesmen will slowly push the norms towards being more scammy over generations, and you’d end up in the equilibrium where nearly every salesman is scammy.
Come to think of it, I think I would estimate that ~1% of academics are explicitly thinking about how to further their own career at the cost of science (in ways that are different from imitation).
And how many if you didn’t intervene?
How do you reconcile this with the immediately prior sentence?
Significantly more, maybe 20. To do a proper estimate I’d need to know which field we’re considering, what the base rates are, etc. The thing I should have said was that I expect it makes it ~10x less likely that you become a professor; that seems more robust to the choice of field and isn’t conditional on base rates that I don’t know.
The Internet suggests a base rate of 3-5%, which means without intervention 3-5 of them would become professors; if that’s true I would say that with intervention an expected 0.4 of them would become professors.
I didn’t mean that it was literally impossible for a person who doesn’t follow the incentives to get into academia, I meant that it was much less likely. I do in fact know people in academia who I think are reasonably good at not following bad incentives.