I think it’s empirical observation. Goodhart looked around, saw in many domains that U diverged from V in a bad way after it became a tracked metric, while seeing no examples of U diverging from a theoretical V’ in a good way, and then minted the “law.”
Upon further analysis, no-one has come up with a counterexample not already covered by the built in exceptions (if U is sufficiently close to V, then maximizing U is fine—eg Moneyball; OR if there is relatively low benefit to perform, agents won’t attempt to maximize U—eg anything using Age as U like senior discounts or school placements)
The world doesn’t just happen to behave in a certain way. The probability that all examples point in a single direction without some actual mechanism causing it is negligible.
I think it’s empirical observation. Goodhart looked around, saw in many domains that U diverged from V in a bad way after it became a tracked metric, while seeing no examples of U diverging from a theoretical V’ in a good way, and then minted the “law.” Upon further analysis, no-one has come up with a counterexample not already covered by the built in exceptions (if U is sufficiently close to V, then maximizing U is fine—eg Moneyball; OR if there is relatively low benefit to perform, agents won’t attempt to maximize U—eg anything using Age as U like senior discounts or school placements)
The world doesn’t just happen to behave in a certain way. The probability that all examples point in a single direction without some actual mechanism causing it is negligible.