By Goodhart’s law, this set has the property that any will with probability 1 be uncorrelated with outside the observed domain.
If we have a collection of variables , and , then is positively correlated in practice with most expressed simply in terms of the variables.
I’ve seen Goodhart’s law as an observation or a fact of human society—you seem to have a mathematical version of it in mind. Is there a reference for that.
It seems I didn’t articulate my point clearly. What I was saying is that V and V’ are equally hard to define, yet we all assume that true human values has a Goodhart problem (rather than a reverse Goodhart problem). This can’t be because of the complexity (since the complexity is equal) nor because we are maximising a proxy (because both have the same proxy).
So there is something specific about (our knowledge of) human values which causes us to expect Goodhart problems rather than reverse Goodhart problems. It’s not too hard to think of plausible explanations (fragility of value can be re-expressed in terms of simple underlying variables to get results like this), but it does need explaining. And it might not always be valid (eg if we used different underlying variables, such as the smooth-mins of the ones we previously used, then fragility of value and Goodhart effects are much weaker), so we may need to worry about them less in some circumstances.
It seems I didn’t articulate my point clearly. What I was saying is that V and V’ are equally hard to define, yet we all assume that true human values has a Goodhart problem (rather than a reverse Goodhart problem). This can’t be because of the complexity (since the complexity is equal) nor because we are maximising a proxy (because both have the same proxy).
So there is something specific about (our knowledge of) human values which causes us to expect Goodhart problems rather than reverse Goodhart problems. It’s not too hard to think of plausible explanations (fragility of value can be re-expressed in terms of simple underlying variables to get results like this), but it does need explaining. And it might not always be valid (eg if we used different underlying variables, such as the smooth-mins of the ones we previously used, then fragility of value and Goodhart effects are much weaker), so we may need to worry about them less in some circumstances.
Thanks for writing this.
For myself, I know that power dynamics are important, but I’ve chosen to specialise down on the “solve technical alignment problem towards a single entity” and leave those multi-agent concerns to others (eg the GovAI part of the FHI), except when they ask for advice.
V and V’ are symmetric; indeed, you can define V as 2U-V’. Given U, they are as well defined as each other.
The idea that maximising the proxy will inevitably end up reducing the true utility seems a strong implicit part of Goodharting the way it’s used in practice.
After all, if the deviation is upwards, Goodharting is far less of a problem. It’s “suboptimal improvement” rather than “inevitable disaster”.
Ah, understood. And I think I agree.
SIA is the Bayesian update on knowing your existence (ie if they were always going to ask if dadadarren existed, and get a yes or no answer). The other effects come from issues like “how did they learn of your existence, and what else could they have learnt instead?” This often does change the impact of learning facts, but that’s not a specifically anthropics problem.
Depends; when you constructed your priors, did you already take that fact into explicit account? You can “know” things, but not have taken them into account.
So regardless of how we describe the difference between T1 and T2, SIA will definitely think that T1 is a lot more likely once we start colonising space, if we ever do that.
SIA isn’t needed for that; standard probability theory will be enough (as our becoming grabby is evidence that grabbiness is easier than expected, and vice-versa).
I think there’s a confusion with SIA and reference classes and so on. If there are no other exact copies of me, then SIA is just standard Bayesian update on the fact that I exist. If theory T_i has prior probability p_i and gives a probability q_i of me existing, then SIA changes its probability to q_i*p_i (and renormalises).
Effects that increase the expected number of other humans, other observers, etc… are indirect consequences of this update. So a theory that says life in general is easy also says that me existing is easy, so gets boosted. But “Earth is special” theories also get boosted: if a theory claims life is very easy but only on Earth-like planets, then those also get boosted.
In this process, I never have to consider “this awakening” as a member of any reference class. Do you think “keeping the score” this way invalid?
Different ways of keeping the score give different answers. So, no, I don’t think that’s invalid.
In the classical sleeping beauty problem, if I guess the coin was tails, I will be correct in 50% of the experiments, and in 67% of my guesses. Whether you score by “experiments” or by “guesses” gives a different optimal performance.
I didn’t fully define those theories, and, indeed, if they depended on commonness of life, then SAI would prefer .
But if I posited instead that and differ only in the propensity for aliens to become grabby or not, then SIA would indeed be indifferent between them.
That is the aim. It’s easy to program an AI that doesn’t care too much about the reward signal—the trick is to find a way that it doesn’t care in a specific way that aligns it with our preferences.
eg what would you do if you had been told to maximise some goal, but were told that your reward signal would be corrupted and over-simplified? You can start doing some things in that situation to maximise your chance of not-wireheading; I want to program the AI to do similarly.
Almost equally hard to define. You just need to define U, which, by assumption, is easy.