Your article and Aaronson’s are similar in tone and superficially similar in content, but yours embodies a more specific and more interesting (to me) central idea. To wit, Malthusianisms is basically about Umeshisms, which is really a restatement of the Pareto Principle. Whereas this article is about the Hansonesque idea that success in a domain is not about proficiency in the domain.
The idea seems to be in essence a restatement or reapplication of Campbell’s Law: A system’s metrics and its goals are not equivalent, so successful behavior tends to become deranged. Though they are not quite the same idea.
Not to say that expounding upon the idea is a bad thing. Your examples are quite elegant.
Campbell’s Law posits that a metric is used because it has historically correlated with a difficult-to-measure desirable property; but that it becomes deranged only when it is used to make decisions that the measured people care about.
Historically, better-educated students do well on standardized tests, when those tests don’t matter. But once you enact a test to discriminate amongst students for purposes those students care about (like getting into prestigious colleges), your measurement of academic achievement will be confounded by your measurement of test-taking skills.
Certainly. I think the same principle applies in many of the listed cases, though. Scientific publications in particular likely developed their current standards at least partially because in the past they filtered for genuinely revolutionary results.
One might as easily say that Campbell’s Law is a sub-principle of the observed phenomenon.
Replications make much more sense as an example. You could also add the file-drawer problem in research. Why do we not see studies that do not find anything? Because there is no prestige in publishing them. (Some journals do try to correct for this, but they have to explicitly do that)
Your article and Aaronson’s are similar in tone and superficially similar in content, but yours embodies a more specific and more interesting (to me) central idea. To wit, Malthusianisms is basically about Umeshisms, which is really a restatement of the Pareto Principle. Whereas this article is about the Hansonesque idea that success in a domain is not about proficiency in the domain.
The idea seems to be in essence a restatement or reapplication of Campbell’s Law: A system’s metrics and its goals are not equivalent, so successful behavior tends to become deranged. Though they are not quite the same idea.
Not to say that expounding upon the idea is a bad thing. Your examples are quite elegant.
Campbell’s Law posits that a metric is used because it has historically correlated with a difficult-to-measure desirable property; but that it becomes deranged only when it is used to make decisions that the measured people care about.
Historically, better-educated students do well on standardized tests, when those tests don’t matter. But once you enact a test to discriminate amongst students for purposes those students care about (like getting into prestigious colleges), your measurement of academic achievement will be confounded by your measurement of test-taking skills.
Certainly. I think the same principle applies in many of the listed cases, though. Scientific publications in particular likely developed their current standards at least partially because in the past they filtered for genuinely revolutionary results.
One might as easily say that Campbell’s Law is a sub-principle of the observed phenomenon.
Replications make much more sense as an example. You could also add the file-drawer problem in research. Why do we not see studies that do not find anything? Because there is no prestige in publishing them. (Some journals do try to correct for this, but they have to explicitly do that)