Nice point. I think most of the time this doesn’t apply, because agents don’t just try to maximize “utility” as some abstract quantity, but possess (transparently or opaquely) some function which they concretely try to increase. Utility is a device, specific to each agent, that refers to states of the world, and isn’t a kind of universal brownie point that everyone likes.
On the other hand, putting “utility” as your utility is fertile ground for wireheading temptations. I define my utility to be ! There, I win.
Thanks. I made some quick changes but it’s probably not structured enough for your taste. Ah well