You could regard carefully controlling one’s expression of one’s utility function as being like a vote, and so subject to that blight of strategic voting, in general people have an incentive to understate their preferences about scenarios they consider unlikely/vice versa, which influences the probability of those outcomes in unpredictable ways and fouls their strategy, or to understate valuations when buying and overstate when selling, this may add up to a game that cannot be played well, a coordination problem, outcomes no one wanted.
But I don’t think humans are all that guileful about how they express their utility function. Most of them have never actually expressed a utility function before, it’s not easy to do, it’s not like checking a box on a list of 20 names. People know it’s a game that can barely be played even in ordinary friendships, people don’t know how to lie strategically about their preferences to the youtube recommender system, let alone their neural lace.
1: The best approach to aggregating preferences doesn’t involve voting systems.
You could regard carefully controlling one’s expression of one’s utility function as being like a vote, and so subject to that blight of strategic voting, in general people have an incentive to understate their preferences about scenarios they consider unlikely/vice versa, which influences the probability of those outcomes in unpredictable ways and fouls their strategy, or to understate valuations when buying and overstate when selling, this may add up to a game that cannot be played well, a coordination problem, outcomes no one wanted.
But I don’t think humans are all that guileful about how they express their utility function. Most of them have never actually expressed a utility function before, it’s not easy to do, it’s not like checking a box on a list of 20 names. People know it’s a game that can barely be played even in ordinary friendships, people don’t know how to lie strategically about their preferences to the youtube recommender system, let alone their neural lace.