Some general remarks on incomparability: Human preferences may be best modelled not by a total ordering but by a preorder, which means that when you compare A and B there are four different possible outcomes:
A is better than B.
B is better than A.
A and B are exactly equally good.
A and B are incomparable.
and these last two are not at all the same thing. In particular, if A and B are incomparable it doesn’t follow that anything better than A is also better than B.
(I don’t think this is saying anything you haven’t already figured out, but you may be glad to know that such ideas are already out there.)
John Conway’s beautiful theory of combinatorial games puts a preorder relation on games, which I’ve often thought might be a useful model for human preferences. (It’s related to his theory of “surreal numbers”, which might also be useful; e.g., its infinities and infinitesimals provide a way, should we want one, of representing situations where one thing is literally infinitely more important than another, even though the other still matters a nonzero amount.) Here, it turns out that the right notion of equality is “if you combine game A with a role-reversed version of game B in a certain way, the resulting game is always won by whoever moves second” whereas incomparability is ”… the resulting game is always won by whoever moves first”; if two games are equal then giving either player in the combined game a tiny advantage makes them win, but if two games are incomparable then there may be a substantial range of handicaps you can give either player while still leaving the resulting game a first-player win and hence leaving the two games incomparable.
On the specific question of multiple utility functions: It’s by no means the only way to get this sort of incomparability, but I agree that something of the kind is probably going on in human brains: two subsystems reporting preferences that pull in different directions, and no stable and well defined way to adjudicate between them.
[EDITED to add:] But I agree with ChaosMote that actually “utility functions” may not be the best name for these things, not only for ChaosMote’s reason that our behaviour may not be well modelled as maximizing anything but also because maybe it’s best to reserve the term “utility function” for something that attempts to describe overall preferences.
Some general remarks on incomparability: Human preferences may be best modelled not by a total ordering but by a preorder, which means that when you compare A and B there are four different possible outcomes:
A is better than B.
B is better than A.
A and B are exactly equally good.
A and B are incomparable.
and these last two are not at all the same thing. In particular, if A and B are incomparable it doesn’t follow that anything better than A is also better than B.
(I don’t think this is saying anything you haven’t already figured out, but you may be glad to know that such ideas are already out there.)
John Conway’s beautiful theory of combinatorial games puts a preorder relation on games, which I’ve often thought might be a useful model for human preferences. (It’s related to his theory of “surreal numbers”, which might also be useful; e.g., its infinities and infinitesimals provide a way, should we want one, of representing situations where one thing is literally infinitely more important than another, even though the other still matters a nonzero amount.) Here, it turns out that the right notion of equality is “if you combine game A with a role-reversed version of game B in a certain way, the resulting game is always won by whoever moves second” whereas incomparability is ”… the resulting game is always won by whoever moves first”; if two games are equal then giving either player in the combined game a tiny advantage makes them win, but if two games are incomparable then there may be a substantial range of handicaps you can give either player while still leaving the resulting game a first-player win and hence leaving the two games incomparable.
On the specific question of multiple utility functions: It’s by no means the only way to get this sort of incomparability, but I agree that something of the kind is probably going on in human brains: two subsystems reporting preferences that pull in different directions, and no stable and well defined way to adjudicate between them.
[EDITED to add:] But I agree with ChaosMote that actually “utility functions” may not be the best name for these things, not only for ChaosMote’s reason that our behaviour may not be well modelled as maximizing anything but also because maybe it’s best to reserve the term “utility function” for something that attempts to describe overall preferences.