Let me try to repair Goodhart’s law to avoid these problems:
By statistics, we should very generally expect two random variables to be uncorrelated unless there’s a “good reason” to expect them to be correlated. Goodhart’s law says that if U and V are correlated in some distribution, then (1) if a powerful optimizer tries to maximize U, then it will by default go far out of the distribution, (2) the mere fact that U and V were correlated in the distribution does not in itself constitute a “good reason” to expect them to be correlated far out of the distribution, so by default they won’t be; (3) therefore we expect Goodhart’s law “by default”: you optimize U, thus go out of the distribution, thus break the correlation between U and V, and then V regresses back down to its mean.
So then we can start going through examples:
GDP vs human flourishing: This example fits all the defaults. There is no “good reason” to expect an extremely-out-of-distribution correlation between “GDP” and “human flourishing”—really the only reason to expect a correlation is the fact that they’re correlated in-distribution, and by itself that’s not enough to count as a “good reason”. And we definitely expect that powerfully maximizing GDP would push it far out-of-distribution. Therefore we expect Goodhart’s law—if you maximize GDP hard enough, then human flourishing will stop going up and start going down as it regresses to the mean.
GDP vs “twice GDP minus human flourishing”: Here there is a very good a priori reason to expect an extremely-out-of-distribution correlation between the two sides—namely the fact that “GDP” is part of both. So the default expectation doesn’t apply.
GDP vs log(GDP): Here there’s an even more obvious, a priori reason to expect a robust correlation across all possible configurations of matter in all possible universes. So the default expectation doesn’t apply.
“Mass of an object” vs “total number of protons and neutrons in the object”: The default expectation that “optimization takes you far out of the distribution” doesn’t really apply here, because regularities hold in a much broader “distribution” if the regularity comes from basic laws of physics, rather than from regularities concerning human-sized objects and events. So you can have a quite powerful optimization process trying to maximize an object’s mass, yet stay well within the distribution of environments where this particular correlation remains robust. (A powerful enough optimizer could eventually make a black hole, which would indeed break this correlation, and then we get Goodhart’s law. Other physics-derived correlations would be truly unbreakable though, like inertial mass vs gravitational mass.)
“The utility of the worst-off human” vs “The utility of the average human”: Is there a “good reason” to expect these to be correlated extremely-out-of-distribution? Yes! Mathematically, if the former goes to infinity, then the latter has to go to infinity too. So we have a sound a priori reason to at least question the Goodhart’s law default. We need a more object-level analysis to decide what would happen.
Let me try to repair Goodhart’s law to avoid these problems:
By statistics, we should very generally expect two random variables to be uncorrelated unless there’s a “good reason” to expect them to be correlated. Goodhart’s law says that if U and V are correlated in some distribution, then (1) if a powerful optimizer tries to maximize U, then it will by default go far out of the distribution, (2) the mere fact that U and V were correlated in the distribution does not in itself constitute a “good reason” to expect them to be correlated far out of the distribution, so by default they won’t be; (3) therefore we expect Goodhart’s law “by default”: you optimize U, thus go out of the distribution, thus break the correlation between U and V, and then V regresses back down to its mean.
So then we can start going through examples:
GDP vs human flourishing: This example fits all the defaults. There is no “good reason” to expect an extremely-out-of-distribution correlation between “GDP” and “human flourishing”—really the only reason to expect a correlation is the fact that they’re correlated in-distribution, and by itself that’s not enough to count as a “good reason”. And we definitely expect that powerfully maximizing GDP would push it far out-of-distribution. Therefore we expect Goodhart’s law—if you maximize GDP hard enough, then human flourishing will stop going up and start going down as it regresses to the mean.
GDP vs “twice GDP minus human flourishing”: Here there is a very good a priori reason to expect an extremely-out-of-distribution correlation between the two sides—namely the fact that “GDP” is part of both. So the default expectation doesn’t apply.
GDP vs log(GDP): Here there’s an even more obvious, a priori reason to expect a robust correlation across all possible configurations of matter in all possible universes. So the default expectation doesn’t apply.
“Mass of an object” vs “total number of protons and neutrons in the object”: The default expectation that “optimization takes you far out of the distribution” doesn’t really apply here, because regularities hold in a much broader “distribution” if the regularity comes from basic laws of physics, rather than from regularities concerning human-sized objects and events. So you can have a quite powerful optimization process trying to maximize an object’s mass, yet stay well within the distribution of environments where this particular correlation remains robust. (A powerful enough optimizer could eventually make a black hole, which would indeed break this correlation, and then we get Goodhart’s law. Other physics-derived correlations would be truly unbreakable though, like inertial mass vs gravitational mass.)
“The utility of the worst-off human” vs “The utility of the average human”: Is there a “good reason” to expect these to be correlated extremely-out-of-distribution? Yes! Mathematically, if the former goes to infinity, then the latter has to go to infinity too. So we have a sound a priori reason to at least question the Goodhart’s law default. We need a more object-level analysis to decide what would happen.
Cheers, these are useful classifications.