Interpersonal utility comparisons are not a natural part of utility theory the same way individual utility functions are. I think of them as being important for two reasons:
1: Our own personal moral reasoning does something like interpersonal utility comparisons, and we want to try to formalize that.
2: we want to cooperate with other people to achieve some goal that will benefit all, but in order to define our collective goals we need some form of interpersonal utility comparison.
2.5: We’re about to build an AI and want to program in by hand how it should weigh human values (warning: don’t do 2.5).
Interpersonal utility comparisons are not a natural part of utility theory the same way individual utility functions are. I think of them as being important for two reasons:
1: Our own personal moral reasoning does something like interpersonal utility comparisons, and we want to try to formalize that.
2: we want to cooperate with other people to achieve some goal that will benefit all, but in order to define our collective goals we need some form of interpersonal utility comparison.
2.5: We’re about to build an AI and want to program in by hand how it should weigh human values (warning: don’t do 2.5).