It seems that EBM defaults to a do-ocracy. This is alluded to in the discussion of purchasing authorship in prestigious papers. The system seems to work by offering to do all the “hard parts” of publishing. This meshes with Goodheart’s law: figure out which proxy measures people care about, and then game the proxies. We have 3 sets of interests: the organization, the customers of the organization, and the professionals the organization partners with. The customers care about whichever proxy measures they have learned to care about, headline statistics, good words like “randomized controlled trials”, “meta-analysis” etc (see the Bingo Card Fallacy). The professionals care about prestige. The organization buys associations with the professionals and then sells them to their customers. They smooth away all the frictional costs associated with this transaction and thus gain access to monkeying around with the methods of that transaction in a way that optimizes for the proxies their customers respond to.
There winds up being an ongoing battle. Educators try to teach the public about new, harder-to-game proxies (forest plots! preregistration! and on the very cutting edge, specification curves!*) Organizations have a very large incentive to find ways to game the new proxies. It is actually surprising that much progress is made. The incentives are asymmetric, especially money wise. Good conceptual handles/frames (like the spread of the pyramid of evidence) seem to help a lot. Vox.com recently did an article explaining forest plots, though I am having difficulty finding it now.
The general pattern is that the new proxies need to better along some understandable numerical (meta-anlysis = bigger n!) or graphical dimension (forest plots give an intuitive overview of an entire line of study at a glance). I am hopeful for specification curves for exactly this reason since the output is intuitively graspable. It is then plausible for a meme to spread that evidence without X backing it is lower quality, where X is simply the checklist of hardest to fake proxies for rigor/validity.
One objection might be that you can only build so much on the top of the pyramid of evidence when the base is rotting (poor individual study design), but I think it helps. When Cochrane does a meta-analysis and only includes 10% of the papers in a given research area because the other 90% are crap, this sends a signal to the system. It would be nice if the signal was more robustly responded to of course, but at least it exists.
*Skip to this point to see what the output of a specification curve would be: https://youtu.be/g75jstZidX0?t=1484
The second curve shown illustrates how discontinuities can highlight problems/areas of interest in the specification space.
It seems that EBM defaults to a do-ocracy. This is alluded to in the discussion of purchasing authorship in prestigious papers. The system seems to work by offering to do all the “hard parts” of publishing. This meshes with Goodheart’s law: figure out which proxy measures people care about, and then game the proxies. We have 3 sets of interests: the organization, the customers of the organization, and the professionals the organization partners with. The customers care about whichever proxy measures they have learned to care about, headline statistics, good words like “randomized controlled trials”, “meta-analysis” etc (see the Bingo Card Fallacy). The professionals care about prestige. The organization buys associations with the professionals and then sells them to their customers. They smooth away all the frictional costs associated with this transaction and thus gain access to monkeying around with the methods of that transaction in a way that optimizes for the proxies their customers respond to.
There winds up being an ongoing battle. Educators try to teach the public about new, harder-to-game proxies (forest plots! preregistration! and on the very cutting edge, specification curves!*) Organizations have a very large incentive to find ways to game the new proxies. It is actually surprising that much progress is made. The incentives are asymmetric, especially money wise. Good conceptual handles/frames (like the spread of the pyramid of evidence) seem to help a lot. Vox.com recently did an article explaining forest plots, though I am having difficulty finding it now.
The general pattern is that the new proxies need to better along some understandable numerical (meta-anlysis = bigger n!) or graphical dimension (forest plots give an intuitive overview of an entire line of study at a glance). I am hopeful for specification curves for exactly this reason since the output is intuitively graspable. It is then plausible for a meme to spread that evidence without X backing it is lower quality, where X is simply the checklist of hardest to fake proxies for rigor/validity.
One objection might be that you can only build so much on the top of the pyramid of evidence when the base is rotting (poor individual study design), but I think it helps. When Cochrane does a meta-analysis and only includes 10% of the papers in a given research area because the other 90% are crap, this sends a signal to the system. It would be nice if the signal was more robustly responded to of course, but at least it exists.
*Skip to this point to see what the output of a specification curve would be: https://youtu.be/g75jstZidX0?t=1484 The second curve shown illustrates how discontinuities can highlight problems/areas of interest in the specification space.
note:
“He entered one day the board room”
should be “he entered the board room one day”
“This is not what I thought medicine would be about, let along EBM.”
should be “alone”
“EBM should still be possible to practice anywhere, somewhere-- this remains a worthwhile goal”
reads less awkwardly with somewhere and anywhere reversed (English colloquialism).