Here’s an example use-case where it matters a lot.
Suppose I’m trying to do a typical Progress Studies thing; I look at a bunch of historical examples of major discoveries and inventions, in hopes of finding useful patterns to emulate. For purposes of my data-gathering, I want to do the sort of credit assignment the post talks about: I want to filter out the parade-leaders, because I expect they’re mostly dominated by noise.
I do think you’re mostly-right for purposes of a researcher rationally optimizing for their own status. But that’s a quite different use-case from an external observer trying to understand what factors drive progress, or even a researcher trying to understand key factors in order to boost their own work.
The counterfactual impact of a researcher is analogous to the insight that professional baseball players are largely interchangeable because they are all already selected from the extreme tail of baseball playing ability, which is to say the counterfactual impact of a given player added to the team is also low.
Of course in Moneyball they used this to get good-enough talent within budget, which is not the same as the researcher case. All of fantasy sports is exactly a giant counterfactual exercise; I wonder how far we could get with ‘fantasy labs’ or something.
One way to identify counterfactually-excellent researchers would be to compare the magnitude of their “greatest achievement” and secondary discoveries, because the credit that parade leaders get is often useful for propagating their future success and the people who do more with that boost are the ones who should be given extra credit for originality (their idea) as opposed to novelty (their idea first). Newton and Leibniz both had remarkably successful and diverse achievements, which suggests that they were relatively high in counterfactual impact in most (if not all) of those fields. Another approach would consider how many people or approaches to a problem had tried and failed to solve it: crediting the zeitgeist rather than Newton and/or Leibniz specifically seems to miss a critical question, namely that if neither of them solved it, would it have taken an additional year, or more like 10 to 50? In their case, we have a proxy to an answer: ideas took months or years to spread at all beyond the “centers of discovery” at the time, and so although they clearly took only a few months or years to compete for the prize of first (and a few decades to argue over it), we can relatively safely conjecture that whichever anonymous contender is third in the running is likely to have been behind on at least that timescale. That should be considered in contrast to Andrew Wiles, whose proof of Rermat’s Last Theorem was efficiently and immediately published (and patched as needed) This is also important because other and in particular later luminaries of the field (e.g. Mengoli, Mercator, various Bernoullis, Euler, etc.) might not have had the vocabulary necessary to make as many discoveries as quickly as they did or communicate those discoveries as effectively if not for Newton & Leibniz’s timely contributions.
Here’s an example use-case where it matters a lot.
Suppose I’m trying to do a typical Progress Studies thing; I look at a bunch of historical examples of major discoveries and inventions, in hopes of finding useful patterns to emulate. For purposes of my data-gathering, I want to do the sort of credit assignment the post talks about: I want to filter out the parade-leaders, because I expect they’re mostly dominated by noise.
I do think you’re mostly-right for purposes of a researcher rationally optimizing for their own status. But that’s a quite different use-case from an external observer trying to understand what factors drive progress, or even a researcher trying to understand key factors in order to boost their own work.
A sports analogy is Moneyball.
The counterfactual impact of a researcher is analogous to the insight that professional baseball players are largely interchangeable because they are all already selected from the extreme tail of baseball playing ability, which is to say the counterfactual impact of a given player added to the team is also low.
Of course in Moneyball they used this to get good-enough talent within budget, which is not the same as the researcher case. All of fantasy sports is exactly a giant counterfactual exercise; I wonder how far we could get with ‘fantasy labs’ or something.
One way to identify counterfactually-excellent researchers would be to compare the magnitude of their “greatest achievement” and secondary discoveries, because the credit that parade leaders get is often useful for propagating their future success and the people who do more with that boost are the ones who should be given extra credit for originality (their idea) as opposed to novelty (their idea first). Newton and Leibniz both had remarkably successful and diverse achievements, which suggests that they were relatively high in counterfactual impact in most (if not all) of those fields. Another approach would consider how many people or approaches to a problem had tried and failed to solve it: crediting the zeitgeist rather than Newton and/or Leibniz specifically seems to miss a critical question, namely that if neither of them solved it, would it have taken an additional year, or more like 10 to 50? In their case, we have a proxy to an answer: ideas took months or years to spread at all beyond the “centers of discovery” at the time, and so although they clearly took only a few months or years to compete for the prize of first (and a few decades to argue over it), we can relatively safely conjecture that whichever anonymous contender is third in the running is likely to have been behind on at least that timescale. That should be considered in contrast to Andrew Wiles, whose proof of Rermat’s Last Theorem was efficiently and immediately published (and patched as needed) This is also important because other and in particular later luminaries of the field (e.g. Mengoli, Mercator, various Bernoullis, Euler, etc.) might not have had the vocabulary necessary to make as many discoveries as quickly as they did or communicate those discoveries as effectively if not for Newton & Leibniz’s timely contributions.