Consider a problem with scarce resource allocation in a small community with an appointed decision-taker: utility function that intuitively should be used has utility functions of different members of a community taken with some commensurable coefficients.
So you would need some additional structure to meaningfully combine utility functions, but, depending on a scenario, there often are solutions that seem natural in their domain. Of course, if we extend them beyond their natural domain we get all the weirdness explored by scenarios of Dr. Evil running a trillion simulations of himself to foil CEV-performing Friendly AI, and, arguably, SIA vs SSA paradoxes too.
Consider a problem with scarce resource allocation in a small community with an appointed decision-taker: utility function that intuitively should be used has utility functions of different members of a community taken with some commensurable coefficients.
So you would need some additional structure to meaningfully combine utility functions, but, depending on a scenario, there often are solutions that seem natural in their domain. Of course, if we extend them beyond their natural domain we get all the weirdness explored by scenarios of Dr. Evil running a trillion simulations of himself to foil CEV-performing Friendly AI, and, arguably, SIA vs SSA paradoxes too.