Can you give other conceptions of “impact” that people have proposed, and compare/contrast them with “How does this change my ability to get what I want?”
The next post will cover this.
(no way to double quote it seems...maybe nested BBCode?)
Anyhow, looking forward to that as I was struggling a bit with the claim cannot be a big deal if it doesn’t impact my getting what I want without being tautological.
Well, the claim is tautological, after all! The problem with the first part of this sequence is that it can seem… obvious… until you realize that almost all prior writing about impact has not even acknowledged that we want the AI to leave us able to get what we want (to preserve our attainable utility). By default, one considers what “big deals” have in common, and then thinks about not breaking vases / not changing too much stuff in the world state. This attractor is so strong that when I say, “wait, maybe it’s not primarily about vases or objects”, it didn’t make sense.
The point of the first portion of the sequence isn’t to amaze people with the crazy surprising insane twists I’ve discovered in what impact really is about—it’s to show how things add up to normalcy, so as to set the stage for a straightforward discussion about one promising direction I have in mind for averting instrumental incentives.
The problem with the first part of this sequence is that it can seem… obvious… until you realize that almost all prior writing about impact has not even acknowledged that we want the AI to leave us able to get what we want (to preserve our attainable utility).
Agreed. This has been my impression from reading previous work on impact.
Let me substantiate my claim a bit with a random sampling; I just pulled up a relative reachability blogpost. From the first paragraph, (emphasis mine)
An incorrect or incomplete specification of the objective can result in undesirable behavior like specification gaming or causing negative side effects. There are various ways to make the notion of a “side effect” more precise – I think of it as a disruption of the agent’s environment that is unnecessary for achieving its objective. For example, if a robot is carrying boxes and bumps into a vase in its path, breaking the vase is a side effect, because the robot could have easily gone around the vase. On the other hand, a cooking robot that’s making an omelette has to break some eggs, so breaking eggs is not a side effect.
But notice now we’re talking about “disruption of the agent’s environment”. Relative reachability is indeed tackling the impact measure problem, so using what we now understand we might prefer to reframe as:
We think about “side effects” when they change our attainable utilities, so they’re really just a conceptual discretization of “things which negatively affect us”. We want the robot to prefer policies which avoid overly changing our attainable utilities. For example, if a robot is carrying boxes and bumps into a vase in its path, breaking the vase is a side effect, because it’s not that easy for us to repair the vase...
(no way to double quote it seems...maybe nested BBCode?)
Anyhow, looking forward to that as I was struggling a bit with the claim cannot be a big deal if it doesn’t impact my getting what I want without being tautological.
Well, the claim is tautological, after all! The problem with the first part of this sequence is that it can seem… obvious… until you realize that almost all prior writing about impact has not even acknowledged that we want the AI to leave us able to get what we want (to preserve our attainable utility). By default, one considers what “big deals” have in common, and then thinks about not breaking vases / not changing too much stuff in the world state. This attractor is so strong that when I say, “wait, maybe it’s not primarily about vases or objects”, it didn’t make sense.
The point of the first portion of the sequence isn’t to amaze people with the crazy surprising insane twists I’ve discovered in what impact really is about—it’s to show how things add up to normalcy, so as to set the stage for a straightforward discussion about one promising direction I have in mind for averting instrumental incentives.
Agreed. This has been my impression from reading previous work on impact.
Let me substantiate my claim a bit with a random sampling; I just pulled up a relative reachability blogpost. From the first paragraph, (emphasis mine)
But notice now we’re talking about “disruption of the agent’s environment”. Relative reachability is indeed tackling the impact measure problem, so using what we now understand we might prefer to reframe as: