Short response: I think unitary utility functions are a distraction and don’t describe the decision making of real world intelligent systems very well.
Longer response:
In an environment with a common, scarce and fungible resource that an agent has monotonically nondecreasing preferences over, an agent that is inexploitable with respect to that resource behaves as an expected utility maximiser.
However, the relevant theorems assume that:
The agent has complete preferences: for any pair of options (lotteries in VNM) the agent prefers one of them or is indifferent.
Alternatively, there exists a total order over their preferences
The agent’s preferences are path independent/don’t have any internal state: the option the agent prefers in a particular scenario does not depend on how the agent got there
Consider that human preferences are inherently contextual; what option we prefer in general depends on our history (previous choices, previous experiences, etc.) and the context of the situation
The agent’s preferences are static: they do not change over time
These preconditions are pretty unrealistic and do not describe humans or financial markets well. I do not expect them to describe any generally capable systems in the real world well either. I.e. I suspect that expected utility maximising is anti-natural to general capabilities.
Shard theory presents a compelling rebuttal to expected utility maximisation.
See “Why The Focus on Expected Utility Maximisers”.
Short response: I think unitary utility functions are a distraction and don’t describe the decision making of real world intelligent systems very well.
Longer response: In an environment with a common, scarce and fungible resource that an agent has monotonically nondecreasing preferences over, an agent that is inexploitable with respect to that resource behaves as an expected utility maximiser.
However, the relevant theorems assume that:
The agent has complete preferences: for any pair of options (lotteries in VNM) the agent prefers one of them or is indifferent.
Alternatively, there exists a total order over their preferences
The agent’s preferences are path independent/don’t have any internal state: the option the agent prefers in a particular scenario does not depend on how the agent got there
Consider that human preferences are inherently contextual; what option we prefer in general depends on our history (previous choices, previous experiences, etc.) and the context of the situation
The agent’s preferences are static: they do not change over time
These preconditions are pretty unrealistic and do not describe humans or financial markets well. I do not expect them to describe any generally capable systems in the real world well either. I.e. I suspect that expected utility maximising is anti-natural to general capabilities.
Shard theory presents a compelling rebuttal to expected utility maximisation.