Your brain is conditioning you [...] toward proxies
Because proxies are always leaky, your brain is conditioning you wrong.
I think this is overly pessimistic: humans are pretty functional which is evidence that the brain figured out how to condition towards the real thing or that proxies are actually fine for the most part.
People are quick to warn about Goodhart and point out the various issues with the brain, but what about all the stuff it gets right? It would be interesting to get a rough ratio of useful VS non-useful proxies here.
The problem is, evolution generally doesn’t build in large buffers. Human brains are “pretty functional” in the sense that they just barely managed to be adequate to the challenges that we faced in the ancestral environment. Now that we are radically changing that environment, the baseline “barely adequate” doesn’t have to degrade very much at all before we have concerningly high rates of stuff like obesity, depression, schizophrenia, etc.
(There are other larger problems, but this is a first gentle gesture in the direction of “I think your point is sound but still not reassuring.” I agree you could productively make a list of proxies that are still working versus ones that aren’t holding up in the modern era.)
I think this is overly pessimistic: humans are pretty functional which is evidence that the brain figured out how to condition towards the real thing or that proxies are actually fine for the most part.
People are quick to warn about Goodhart and point out the various issues with the brain, but what about all the stuff it gets right? It would be interesting to get a rough ratio of useful VS non-useful proxies here.
The problem is, evolution generally doesn’t build in large buffers. Human brains are “pretty functional” in the sense that they just barely managed to be adequate to the challenges that we faced in the ancestral environment. Now that we are radically changing that environment, the baseline “barely adequate” doesn’t have to degrade very much at all before we have concerningly high rates of stuff like obesity, depression, schizophrenia, etc.
(There are other larger problems, but this is a first gentle gesture in the direction of “I think your point is sound but still not reassuring.” I agree you could productively make a list of proxies that are still working versus ones that aren’t holding up in the modern era.)