If you update your utility every time new information comes in, the utility is time-inconsistent. This lets you be money-pumped. Hence it’s the kind of thing you would get rid of at you next self-improvement.
The utility function is always the same in this kind of scenario—and is not “updated”.
It typically says something roughly like: stop button not pressed: business as normal—stop button pressed: let the engineeres dismantle your brain. That doesn’t really let you be money-pumped because—for one thing, a pump needs repeated cycles to do much work. Also, after being switched off the agent can’t engage in any economic activities.
Agent’s won’t get rid of such stipulations as they self-improve—under the assumption that a self-improving agent successfully preserves its utility function. Changing the agent’s utility function would typically be very bad—from the point of view of the agent.
If you update your utility every time new information comes in, the utility is time-inconsistent. This lets you be money-pumped. Hence it’s the kind of thing you would get rid of at you next self-improvement.
The utility function is always the same in this kind of scenario—and is not “updated”.
It typically says something roughly like: stop button not pressed: business as normal—stop button pressed: let the engineeres dismantle your brain. That doesn’t really let you be money-pumped because—for one thing, a pump needs repeated cycles to do much work. Also, after being switched off the agent can’t engage in any economic activities.
Agent’s won’t get rid of such stipulations as they self-improve—under the assumption that a self-improving agent successfully preserves its utility function. Changing the agent’s utility function would typically be very bad—from the point of view of the agent.