Okay, I think I disagree that extrapolating beyond the range of your data is Goodharting. I use the term for the narrower case where either the signal or the value stays in the trained range, but become very divergent from each other. E.g. artificial sweeteners break the link between sweetness and calories.
I don’t think this is quite isomorphic to the first paragraph, but highly related: I think of sweetness as a proxy for calories. Are you defining sweetness as a proxy for good for me?
I am thinking of sugar as a proxy for good for me.
I do not think that all instances of training data not matching the environment you are optimizing are Goodhart. However if the reason that the environment does not match the training is because the proxy is large, and the reason the proxy is large is because you are optimizing for it, then the optimization causes the failure of the proxy, which is why I am calling it Goodhart.
Okay, I think I disagree that extrapolating beyond the range of your data is Goodharting. I use the term for the narrower case where either the signal or the value stays in the trained range, but become very divergent from each other. E.g. artificial sweeteners break the link between sweetness and calories.
I don’t think this is quite isomorphic to the first paragraph, but highly related: I think of sweetness as a proxy for calories. Are you defining sweetness as a proxy for good for me?
I am thinking of sugar as a proxy for good for me.
I do not think that all instances of training data not matching the environment you are optimizing are Goodhart. However if the reason that the environment does not match the training is because the proxy is large, and the reason the proxy is large is because you are optimizing for it, then the optimization causes the failure of the proxy, which is why I am calling it Goodhart.