I’m sure professional athletes said the same thing when public metrics began to appear. Generally new signals get push back. There will be winners and losers, and the losers fight back much harder than the winners encourage. In this case the losers would likely be the least epistemically modest of the intellectuals, a particularly nastybunch. But if signals can persist, they get accepted as part of the way things are and life moves on.
This seems to ignore the possibility of flaws in the metrics. For example, if such metrics had been created some time ago, and had included something involving p-values...
There are definitely ways of doing this poorly, but the status quo is really really bad.
In order to create a metric, you should know what the metric is for.
Tenure? I think those metrics already exists. (For better or for worse.)
Citation stats—not a good measure by itself. (It’s independent of both replication and non-replication.)
For one, Expert Political Judgement provided a fair amount of evidence for just how poorly calibrated all famous and well esteemed intellectuals seem to be.
It determined that by asking them to make predictions? If you want good predictors, then ditch the intellectuals—and get good forecasters/forecasts.
This seems to ignore the possibility of flaws in the metrics. For example, if such metrics had been created some time ago, and had included something involving p-values...
In order to create a metric, you should know what the metric is for.
Tenure? I think those metrics already exists. (For better or for worse.)
Citation stats—not a good measure by itself. (It’s independent of both replication and non-replication.)
It determined that by asking them to make predictions? If you want good predictors, then ditch the intellectuals—and get good forecasters/forecasts.