My first reaction (which I still mostly endorse): KISS; OP = EU’ is fine.
I do not share your intuition for criterion 2, because exponentially increasing utility should be achievable, but only because increasing optimization power is, so it seems desirable that constant OP yields subexponential growth in EU, and exponential growth in OP yields exponential growth in EU.
At least for humans, there is an intuitive sign to utility. We wouldn’t say that stubbing your toe is 1,000,000 utils, and getting a car is 1,002,000 utils. It seems to me, especially after reading Omohundro’s “Basic AI Drives”, that there is in some sense an intrinsic zero utility for all OPs.
Disagree. When we talk about the utility of stubbing your toe, we mean the change in utility from what it would have been if you hadn’t stubbed your toe. Obviously this is going to be negative, but that doesn’t put a zero point on your entire utility function.
In your last big paragraph, you seem to be confusing utility of the current world-state with resources with which to increase the utility of future world-states. They are different.
That said, if you need to call some certain amount of utility “zero”, your suggestion of using the utility of whatever would happen without the agent existing does seem to hold merit. (or similarly: expected utility if the agent chooses actions randomly).
Although I still don’t much like OP = EU’/|EU|, it does have some desirable properties. While I don’t like the dependence on origin (singularity at zero = nasty), it nicely breaks the dependence on scale. And I don’t think the presence of absolute values in the equation is all that bad; in fact, it neatly parallels the fact that utility functions can be multiplied only by positive scales while remaining equivalent.
My first reaction (which I still mostly endorse): KISS; OP = EU’ is fine.
I do not share your intuition for criterion 2, because exponentially increasing utility should be achievable, but only because increasing optimization power is, so it seems desirable that constant OP yields subexponential growth in EU, and exponential growth in OP yields exponential growth in EU.
Disagree. When we talk about the utility of stubbing your toe, we mean the change in utility from what it would have been if you hadn’t stubbed your toe. Obviously this is going to be negative, but that doesn’t put a zero point on your entire utility function.
In your last big paragraph, you seem to be confusing utility of the current world-state with resources with which to increase the utility of future world-states. They are different.
That said, if you need to call some certain amount of utility “zero”, your suggestion of using the utility of whatever would happen without the agent existing does seem to hold merit. (or similarly: expected utility if the agent chooses actions randomly).
Although I still don’t much like OP = EU’/|EU|, it does have some desirable properties. While I don’t like the dependence on origin (singularity at zero = nasty), it nicely breaks the dependence on scale. And I don’t think the presence of absolute values in the equation is all that bad; in fact, it neatly parallels the fact that utility functions can be multiplied only by positive scales while remaining equivalent.