Whatever utility calculus you follow, it is a mathematical model.
“All models are false.”
In particular, what’s going wrong here is your model is treating you, the agent, as atomic. In reality, as Kaj Sotala described very well below, you are not an atomic agent, you have an internal architecture, and this architecture has very important ramifications for how you should think about utilities.
If I may make an analogy from the field of AI. In the old days, AI was concerned about something called “discrete search,” which is just a brute force way to look for an optimum in a state space, where each state is essentially an atomic point. Alpha-beta pruning search Deep Blue uses to play chess is an example of discrete search. At some point it was realized that for many problems atomic point-like states resulted in a combinatorial explosion, and in addition states had salient features describable by, say, logical languages. As this realization was implemented, you no longer had a state-as-a-point, but state-as-a-collection-of-logical-statements. And the field of planning was born. Planning has some similarities to discrete search, but because we “opened up” the states into a full blown logical description, the character of the problem is quite different.
If I may be so bold as to summarize this thread:
Whatever utility calculus you follow, it is a mathematical model.
“All models are false.”
In particular, what’s going wrong here is your model is treating you, the agent, as atomic. In reality, as Kaj Sotala described very well below, you are not an atomic agent, you have an internal architecture, and this architecture has very important ramifications for how you should think about utilities.
If I may make an analogy from the field of AI. In the old days, AI was concerned about something called “discrete search,” which is just a brute force way to look for an optimum in a state space, where each state is essentially an atomic point. Alpha-beta pruning search Deep Blue uses to play chess is an example of discrete search. At some point it was realized that for many problems atomic point-like states resulted in a combinatorial explosion, and in addition states had salient features describable by, say, logical languages. As this realization was implemented, you no longer had a state-as-a-point, but state-as-a-collection-of-logical-statements. And the field of planning was born. Planning has some similarities to discrete search, but because we “opened up” the states into a full blown logical description, the character of the problem is quite different.
I think we need to “open up the agent.”