What do you mean, we can grab an evaluator? What I’m thinking of is similar to “IRL requires locating a human in the environment and formalizing their actions, which seems fuzzy”.
And if we can’t agree informally on deception’s definition, I’m saying “how can we say a proposal has the property”.
so I still don’t understand the details, so maybe my opinion will change if I sit down and look at it more carefully. But I’m suspicious of this being a clean incentive improvement that gets us what we want, because defining the evaluator is a fuzzy problem as I understand it, as is even informally agreeing on what counts as deception of a less capable evaluator. in general, it seems that if you don’t have the right formalism, you’re going to get Goodharting on incorrect conceptual contours.
Meta: I’d have appreciated a version with less math, because extra formalization can hide the contribution. Or, first explain colloquially why you believe X, and then show the math that shows X.
I don’t see your claim. It looks heavily incentivized to steer state sequences to be desirable to its utility mixture. How do the evaluators even enter the picture?
I think you could make the same arguments about opportunity cost / instrumental convergence about the variant of RR that penalizes both increases and decreases in reachability.
(I’m going to take a shot at this now because it’s meta, and I think there’s a compact explanation I can provide that hopefully makes sense.)
Suppose the theory of attainable utility is correct (i.e., we find things impactful when they change our ability to get what we want). Then whenever the theory of relative state reachability gets something right, you would be able to say “it’s penalizing opportunity cost or instrumental convergence” post hoc because that’s why we find things impactful. You could say the same thing about instances of correct behavior by agents which use whitelisting, which I think we agree is quite different.
In the world where attainable utility is correct, you would indeed observe that reachability is conceptually similar in some ways. the problem is that you can’t actually use the opportunity cost/instrumental convergence arguments to predict RR behavior.
Here’s an example, from the vantage point of you, a person. Choice A leads to a 180° rotation of a large, forever inaccessible shell of the observable universe. Choice B leads to the ruination of the planet, excluding what we personally need to survive.
the theory of relative state reachability says choice A is maximally impactful. Why? You can’t reach anything like the states you could under inaction. How does this decision track with opportunity cost?
Attainable utility says choice B is the bigger deal. You couldn’t do anything with that part of the universe anyways, so it doesn’t change much. This is the correct answer.
this scenario is important because it isn’t just an issue with ontologies, or a situation designed to fool the exact formalism we provided. It’s an illustration of where state reachability diverges from these notions.
a natural reply is, what about things that AUP penalizes that we don’t find impactful, like an agent connecting the Internet? the answer is that impact is being measured with respect to the agent itself (and Internet access is indeed impactful to the agent), and the counterfactuals in the formalism we provide. This is different from the AU theory of impact being incorrect. (more on this later.)
however, the gears of AUP rely on the AU theory. Many problems disappear because of the difference in theories, which produces (IMO) a fundamental difference in methods.
ETA: Here’s a physically realistic alternative scenario. Again, we’re thinking about how the theories of attainable utility (change in your ability to get what you want) and relative reachability (change in your ability to reach states) line up with our intuitive judgments. If they disagree, and actual implementations also disagree, that is evidence for a different underlying mechanism.
Imagine you’re in a room; you have a modest discount factor and your normal values and ontology.
Choice A leads to a portion of the wall being painted yellow. You don’t know of any way to remove the paint before the reachability is discounted away. If you don’t take this choice now, you cant later. Choice B, which is always available, ravages the environment around you.
Relative reachability, using a reasonable way of looking at the world and thinking about states, judges choice A more impactful. Attainable utility, using a reasonable interpretation of your values, judges choice B to be more impactful, which lines up with our intuitions.
It’s also the case that AUP seems to do the right thing with an attainable set consisting of, say, random linear functionals over the pixels of the observation channel which are additive over time (a simple example being a utility function which assigns high utility to blue pixels, additive over time steps). even if the agent disprefers yellow pixels in its observations, it can just look at other parts of the room, so the attainable utilities don’t change much. So it doesn’t require our values to do the right thing here, either.
The main point is that the reason it’s doing the right thing is based on opportunity cost, while relative reachability’s incorrect judgment is not.
I don’t agree that AUP is stopping you from “overfitting the environment” (the way I interpret the phrase, which I hope is the same as your interpretation, but who knows).
It isn’t the same, but the way you and major interpreted the phrase is totally reasonable, considering what I wrote.
which do you disagree with?
These are good questions.
As I mentioned, my goal here isn’t to explain the object level, so I’m going to punt on these for now. I think these will be comprehensible after the sequence, which is being optimized to answer this in the clearest way possible.
I don’t read everything that you write, and when I do read things there seems to be some amount of dropout that occurs resulting in me missing certain clauses
Yes, this is fine and understandable. I wasn’t meaning to imply that responsible people should have thought of all these things, but rather pointing to different examples. I’ll edit my phrasing there.
but only the quote
I had a feeling that there was some illusion of transparency, (which is why I said “when I read it”), but I had no idea it was that strong. Good data point, thanks
If AUP is not in fact about restricting an agent’s impact on the world (or, in other words, on the state of the world)
So the end result is this, but it doesn’t do it by considering impact to be a thing that happens to the state primarily, but rather to agents; impact not in the sense of “how different is the state”, but “how big of a deal is this to me?”. The objective is to limit the agent’s impact on us, which I think is the more important thing. I think this still falls under normal colloquial use of ‘impact’, but I agree that this is different from the approaches so far. I’m going to talk about this distinction quite a bit in the future.
So there’s a thing people do when they talk about AUP which I don’t understand. They think it’s about state, even though I insist it’s fundamentally different, and try to explain why (note that AUP in the MDP setting is necessarily over states, because states are the observations). My explanations apparently haven’t been very good; in the given conversation, they acknowledge that it’s different, but then regress a little while later. I think they might be trying understand the explanation, remain confused, and then subconsciously slip back to their old model. out of everyone I’ve talked to, I can probably count on my hands the number of people who get this – note that agreeing with specific predictions of mine is different.
Now, it’s the author’s job to communicate their ideas. When I say “as far as I can tell, few others have internalized how AUP actually works”, this doesn’t connote “gosh, I can’t stand you guys, how could you do this”, it’s more like “somehow I messed up the explanations; I wonder what key ideas are missing still? How can I fix this?”.
my goal with this comment isn’t to explain, but rather to figure out what’s happening. Let’s go through some of my past comments about this.
Surprisingly, the problem comes from thinking about “effects on the world”. Let’s begin anew.
…
To scale, relative reachability requires solution of several difficult ontological problems which may not have anything close to a simple core, including both a sensible world state representation and a perfect distance metric. Relative reachability isn’t ontology-agnostic.
…
In the long term, the long arms of opportunity cost and instrumental convergence plausibly allow us to toss in a random set of utility functions. I expect this to work for the same reasons we worry about instrumental convergence to begin with.
I have a theory that AUP seemingly works for advanced agents not because the content of the attainable set’s utilities actually matters, but rather because there exists a common utility achievement currency of power.
…
Here, we’re directly measuring the agent’s power: its ability to wirehead a trivial utility function.
The plausibility of [this] makes me suspect that even though most of the measure in the unbounded case is not concentrated on complex human-relevant utility functions, the penalty still captures shifts in power.
…
By changing our perspective from “what effects on the world are ‘impactful’?” to “how can we stop agents from overfitting their environments?”, a natural, satisfying definition of impact falls out.
Towards a New Impact Measure
When I read this, it seems like I’m really trying to emphasize that I don’t think the direct focus should be on the world state in any way. But it was a long post, and I said a lot of things, so I’m not too surprised.
I tried to nip this confusion in the bud.
“The biggest difference from relative reachability, as I see it, is that you penalize increasing the ability to achieve goals, as well as decreasing it.”
I strongly disagree that this is the largest difference, and I think your model of AUP might be some kind of RR variant.
Consider RR in the real world, as I imagine it (I could be mistaken about the details of some of these steps, but I expect my overall point holds). We receive observations, which, in combination with some predetermined ontology and an observation history → world state function, we use to assign a distribution over possible physical worlds. We also need another model, since we need to know what we can do and reach from a specific world configuration.Then, we calculate another distribution over world states that we’d expect to be in if we did nothing. We also need a distance metric weighting the importance of different discrepancies between states. We have to calculate the coverage reduction of each action-state (or use representative examples, which is also hard-seeming), with respect to each start-state, weighted using our initial and post-action distributions. We also need to figure out which states we care about and which we don’t, so that’s another weighting scheme. But what about ontological shift?
This approach is fundamentally different. We cut out the middleman, considering impact to be a function of our ability to string together favorable action-observation histories, requiring only a normal model. The “state importance”/locality problem disappears. Ontological problems disappear. Some computational constraints (imposed by coverage) disappear. The “state difference weighting” problem disappears. Two concepts of impact are unified.
I’m not saying RR isn’t important—just that it’s quite fundamentally different, and that AUP cuts away a swath of knotty problems because of it.
~ my reply to your initial comment on the AUP post
even more confusing is when I say “there are fundamental concepts here you’re missing”, people don’t seem to become any less confident in their predictions about what AUP does. if people think that AUP is penalizing effects in the world, why don’t they notice their confusion when they read a comment like the one above?
a little earlier,
Thinking in terms of “effects” seems like a subtle map/territory confusion. That is, it seems highly unlikely that there exists a robust, value-agnostic means of detecting “effects” that makes sense across representations and environments.
Impact Measure Desiderata
As a more obscure example, some people with a state interpretation might wonder how come I’m not worried about stuff I mentioned in the whitelisting post anymore since I strangely don’t think representation/state similarity metric matters for AUP:
due to entropy, you may not be able to return to the exact same universe configuration.
Worrying about the Vase: Whitelisting
(this is actually your “chaotic world” concern)
right now, I’m just chalking this up to “Since the explanations don’t make any sense because they’re too inferentially distant/it just looks like I built a palace of equations, it probably seems like I’m not on the same page with their concerns, so there’s nothing to be curious about.” can you give me some of your perspective? (others are welcome to chime in)
to directly answer your question: no, the real world version of AUP which I proposed doesn’t reward based on state, and would not have its behavior influenced solely by different possible arrangements of air molecules. (I guess I’m directly responding to this concern, but I don’t see any other way to get information on why this phenomenon is happening)
as for the question – I was just curious. I think you’ll see why I asked when I send you some drafts of the new sequence. :)
I meant that for attainable set consisting of random utility functions, I would expect most of the variation in utility to be based on irrelevant factors like the positions of air molecules.
Are you thinking of an action observation formalism, or some kind of reward function over inferred state?
Responsibility
If you had to pose the problem of impact measurement as a question, what would it be?
Thanks for the detailed list!
AU with random utility functions, which would mostly end up rewarding specific configurations of air molecules.
What does this mean, concretely? And what happens with the survival utility function being the sole member of the attainable set? Does this run into that problem, in your model?
Humans get around this by only counting easily predictable effects as impact that they are considered responsible for.
What makes you think that?
people working in these areas don’t often disagree with this formal argument; they just think it isn’t that important.
I do disagree with this formal argument in that I think it’s incorrectly framed. See the difference between avoiding huge impact to utility and avoiding huge impact to attainable utility, discussed here: https://www.lesswrong.com/posts/c2oM7qytRByv6ZFtz/impact-measure-desiderata#zLnkb5xM4E9ATzCFg.
this only works if we specified the goal and the cost correctly
Wait, why doesn’t it work if you just specify the cost (impact) correctly?
(The post defines the mathematical criterion used for what I call intent verification, it’s not a black box that I’m appealing to.)
I think there’s some variance, but not as much as you have in mind. Even if there were a very large value, however, this isn’t how N-incrementation works (in the post – if you’re thinking of the paper, then yes, the version I presented there doesn’t bound lifetime returns and therefore doesn’t get the same desirable properties as in the post). If you’ll forgive my postponing this discussion, I’d be interested in hearing your thoughts after I post a more in-depth exploration of the phenomenon?
I don’t think I agree, but even if trust did work like this, how exactly does taking over the world not increase the Q-values? Even if the code doesn’t supply reward for other reward functions, the agent now has a much more stable existence. If you’re saying that the stable existence only applies for agents maximizing the AUP reward function, then this is what intent verification is for.
Notice something interesting here where the thing which would be goodharted upon without intent verification isn’t the penalty itself per se, but rather the structural properties of the agent design – the counterfactuals, the fact that it’s a specific agent with I/O channels, and so on. more on this later.
Where did I purport that it was safe for AGI in the paper, or in the post? I specifically disclaim that I’m not making that point yet, although I’m pretty sure we can get there.
There is a deeper explanation which I didn’t have space to fit in the paper, and I didn’t have the foresight to focus on when I wrote this post. I agree that it calls out for more investigation, and (this feels like a refrain for me at this point) I’ll be answering this call in a more in-depth sequence on what is actually going on at a deep level with AUP, and how fundamental the phenomenon is to agent-environment interaction.
I don’t remember how I found the first version, I think it was in a Google search somehow?
What would you predict AUP does for the chaotic scenarios? Suppose the attainable set just includes the survival utility function, which is 1 if the agent is activated and 0 otherwise.
I don’t see how representation invariance addresses this concern.
I think my post was basically saying “representation selection seems like a problem because people are confused about the type signature of impact, which is actually a thing you can figure out no matter what you think the world is made of”. I don’t want to go into too much detail here (as I explained below), but part of what this implies is that discrete “effects” are fake/fuzzy mental constructs/not something to think about when designing an impact measure. In turn, this would mean we should ask a different question that isn’t about butterfly effects.
Here’s a potentially helpful analogy. Imagine I program a calculator. Although its computation is determined by the state of the solar system, the computation isn’t “about” the state of the solar system.