You have a true goal, V. Then you take the set of all potential proxies that have an observed correlation with V, let’s call this S(V). By Goodhart’s law, this set has the property that any U∈S(V) will with probability 1 be uncorrelated with V outside the observed domain.
Then you can take the set S(2U−V). This set will have the property that any U′∈S(2U−V) will with probability 1 be uncorrelated with 2U−V outside the observed domain. This is Goodhart’s law, and it still applies.
Your claim is that there is one element, U∈S(2U−V) in particular, which will be (positively) correlated with 2U−V. But such proxies still have probability 0. So how is that anti-Goodhart?
Pairing up V and 2U−V to show equivalence of cardinality seems to be irrelevant, and it’s also weird.2U−V is an element of 2S(V)−V, and this depends on V.
By Goodhart’s law, this set has the property that any U∈S(V) will with probability 1 be uncorrelated with V outside the observed domain.
If we have a collection of variables {v}, and V=max(v), then V is positively correlated in practice with most U expressed simply in terms of the variables.
I’ve seen Goodhart’s law as an observation or a fact of human society—you seem to have a mathematical version of it in mind. Is there a reference for that.
I ended up using mathematical language because I found it really difficult to articulate my intuitions. My intuition told me that something like this had to be true mathematically, but the fact that you don’t seem to know about it makes me consider this significantly less likely.
If we have a collection of variables {v}, and V=max(v), then V is positively correlated in practice with most U expressed simply in terms of the variables.
Yes, but V also happens to be very strongly correlated with most U that are equal to V. That’s where you do the cheating. Goodhart’s law, as I understand it, isn’t a claim about any single proxy-goal pair. That would be equivalent to claiming that “there are no statistical regularities, period”. Rather, it’s a claim about the nature of the set of all potential proxies.
In a Bayesian language, Goodhart’s law sets the prior probability of any seemingly good proxy being a good proxy, which is virtually 0. If you have additional evidence, like knowing that your proxy can be expressed in a simple way using your goal, then obviously the probabilities are going to shift.
And that’s how your V and V′ are different. In the case of V, the selection of U is arbitrary. In the case of V′, the selection of U isn’t arbitrary, because it was already fixed when you selected V′. But again, if you select a seemingly good proxy U′ at random, it won’t be an actually good proxy.
You have a true goal, V. Then you take the set of all potential proxies that have an observed correlation with V, let’s call this S(V). By Goodhart’s law, this set has the property that any U∈S(V) will with probability 1 be uncorrelated with V outside the observed domain.
Then you can take the set S(2U−V). This set will have the property that any U′∈S(2U−V) will with probability 1 be uncorrelated with 2U−V outside the observed domain. This is Goodhart’s law, and it still applies.
Your claim is that there is one element, U∈S(2U−V) in particular, which will be (positively) correlated with 2U−V. But such proxies still have probability 0. So how is that anti-Goodhart?
Pairing up V and 2U−V to show equivalence of cardinality seems to be irrelevant, and it’s also weird.2U−V is an element of 2S(V)−V, and this depends on V.
If we have a collection of variables {v}, and V=max(v), then V is positively correlated in practice with most U expressed simply in terms of the variables.
I’ve seen Goodhart’s law as an observation or a fact of human society—you seem to have a mathematical version of it in mind. Is there a reference for that.
I ended up using mathematical language because I found it really difficult to articulate my intuitions. My intuition told me that something like this had to be true mathematically, but the fact that you don’t seem to know about it makes me consider this significantly less likely.
Yes, but V also happens to be very strongly correlated with most U that are equal to V. That’s where you do the cheating. Goodhart’s law, as I understand it, isn’t a claim about any single proxy-goal pair. That would be equivalent to claiming that “there are no statistical regularities, period”. Rather, it’s a claim about the nature of the set of all potential proxies.
In a Bayesian language, Goodhart’s law sets the prior probability of any seemingly good proxy being a good proxy, which is virtually 0. If you have additional evidence, like knowing that your proxy can be expressed in a simple way using your goal, then obviously the probabilities are going to shift.
And that’s how your V and V′ are different. In the case of V, the selection of U is arbitrary. In the case of V′, the selection of U isn’t arbitrary, because it was already fixed when you selected V′. But again, if you select a seemingly good proxy U′ at random, it won’t be an actually good proxy.