Does anyone know if private or corporate research has abandoned NHST and adopted better alternatives?
My suspicion is that the answer is no, because even companies that depend heavily on the correctness of their R&D output (like 3M or Dow Chemical, or marketing measurements at Amazon and Facebook) mostly have the problem that people chosen to do research are chosen by people who don’t know about things like p-hacking. Even if they are diligent and careful business people, I expect a modest epistemology to conclude “do what Nature does” and stick with the current methods.
On the other hand, it seems like skipping the analysis part completely by tightly coupling research outputs to products/practices is a more common strategy, particularly for people like Amazon and Facebook where it can be very rapidly iterated. Yet it really seems like there is an opportunity to consistently win beyond what the market is doing here.
This is causing me to wonder how exactly do go about betting on an institution that placed its bets on better analysis. I feel like in order to be sure these methods are really being put to work, a new institution will have to be built for the purpose. It also seems like some careful consideration needs to be given to which market the institution will operate in, because it should be somewhere the edge provided by better analysis has the greatest impact.
One step beyond this: is inducing mimicry of good practices via effective competition something that has been analyzed in an EA context?
(Sort of a very long delayed followup to https://www.lesswrong.com/posts/ttvnPRTxFyru9Hh2H/against-nhst , tracking down one specific strand of the debate.)
Does anyone know if private or corporate research has abandoned NHST and adopted better alternatives?
My suspicion is that the answer is no, because even companies that depend heavily on the correctness of their R&D output (like 3M or Dow Chemical, or marketing measurements at Amazon and Facebook) mostly have the problem that people chosen to do research are chosen by people who don’t know about things like p-hacking. Even if they are diligent and careful business people, I expect a modest epistemology to conclude “do what Nature does” and stick with the current methods.
On the other hand, it seems like skipping the analysis part completely by tightly coupling research outputs to products/practices is a more common strategy, particularly for people like Amazon and Facebook where it can be very rapidly iterated. Yet it really seems like there is an opportunity to consistently win beyond what the market is doing here.
This is causing me to wonder how exactly do go about betting on an institution that placed its bets on better analysis. I feel like in order to be sure these methods are really being put to work, a new institution will have to be built for the purpose. It also seems like some careful consideration needs to be given to which market the institution will operate in, because it should be somewhere the edge provided by better analysis has the greatest impact.
One step beyond this: is inducing mimicry of good practices via effective competition something that has been analyzed in an EA context?