Nice point. I think most of the time this doesn’t apply, because agents don’t just try to maximize “utility” as some abstract quantity, but possess (transparently or opaquely) some function U(⋅) which they concretely try to increase. Utility is a device, specific to each agent, that refers to states of the world, and isn’t a kind of universal brownie point that everyone likes.
On the other hand, putting “utility” as your utility is fertile ground for wireheading temptations. I define my utility to be ∞! There, I win.
Nice point. I think most of the time this doesn’t apply, because agents don’t just try to maximize “utility” as some abstract quantity, but possess (transparently or opaquely) some function U(⋅) which they concretely try to increase. Utility is a device, specific to each agent, that refers to states of the world, and isn’t a kind of universal brownie point that everyone likes.
On the other hand, putting “utility” as your utility is fertile ground for wireheading temptations. I define my utility to be ∞! There, I win.