The utility function is always the same in this kind of scenario—and is not “updated”.
It typically says something roughly like: stop button not pressed: business as normal—stop button pressed: let the engineeres dismantle your brain. That doesn’t really let you be money-pumped because—for one thing, a pump needs repeated cycles to do much work. Also, after being switched off the agent can’t engage in any economic activities.
Agent’s won’t get rid of such stipulations as they self-improve—under the assumption that a self-improving agent successfully preserves its utility function. Changing the agent’s utility function would typically be very bad—from the point of view of the agent.
The utility function is always the same in this kind of scenario—and is not “updated”.
It typically says something roughly like: stop button not pressed: business as normal—stop button pressed: let the engineeres dismantle your brain. That doesn’t really let you be money-pumped because—for one thing, a pump needs repeated cycles to do much work. Also, after being switched off the agent can’t engage in any economic activities.
Agent’s won’t get rid of such stipulations as they self-improve—under the assumption that a self-improving agent successfully preserves its utility function. Changing the agent’s utility function would typically be very bad—from the point of view of the agent.