Are you losing sleep over the daily deaths in Iraq? Are most LWers? . . . If we cared as much as we signal we do, no one would be able go to work, or post on LW. We’d all be too grief-stricken.
That is exactly what I was talking about when I said “There’s a difference between mental distress and action-motivating desire.”. Utility functions are about choices, not feelings, so I assumed that, in a discussion about utility we would be using the word ‘care’ (as in “If we cared as much as we signal we do”) to refer to motives for action, not mental distress. If this isn’t clear, I’m trying to refer to the same ideas discussed here.
And it also isn’t immediately clear that anyone would really want their utility function to be unbounded (unless I’m misinterpreting the term).
It does not make sense to speak of what someone wants their utility function to be; utility functions just describe actual preferences. Someone’s utility function is unbounded if and only if there are consequences with arbitrarily high utility differences. For every consequence, you can identify one that is over twice as good (relative to some zero point, which can be arbitrarily chosen. This doesn’t really matter if you’re not familiar with the topic, it just corresponds to the fact that if every consequence were 1 utilon better, you would make the same choices because relative utilities would not have changed.) Whether a utility function has this property is important in many circumstances and I consider it an open problem whether humans’ utility functions are unbounded, though some would probably disagree and I don’t know what science doesn’t know.
No, because people are not completely rational. What I ‘really’ want to do is what I would do if I were fully informed, rational, etc. Morality is difficult because our brains do not just tell us what we want. Demonstrated preference would only work with ideal agents, and even then it could only tell you what they want most among the possible options.
That is exactly what I was talking about when I said “There’s a difference between mental distress and action-motivating desire.”. Utility functions are about choices, not feelings, so I assumed that, in a discussion about utility we would be using the word ‘care’ (as in “If we cared as much as we signal we do”) to refer to motives for action, not mental distress. If this isn’t clear, I’m trying to refer to the same ideas discussed here.
It does not make sense to speak of what someone wants their utility function to be; utility functions just describe actual preferences. Someone’s utility function is unbounded if and only if there are consequences with arbitrarily high utility differences. For every consequence, you can identify one that is over twice as good (relative to some zero point, which can be arbitrarily chosen. This doesn’t really matter if you’re not familiar with the topic, it just corresponds to the fact that if every consequence were 1 utilon better, you would make the same choices because relative utilities would not have changed.) Whether a utility function has this property is important in many circumstances and I consider it an open problem whether humans’ utility functions are unbounded, though some would probably disagree and I don’t know what science doesn’t know.
Is this basically saying that you can tell someone else’s utility function by demonstrated preference? It sounds a lot like that.
No, because people are not completely rational. What I ‘really’ want to do is what I would do if I were fully informed, rational, etc. Morality is difficult because our brains do not just tell us what we want. Demonstrated preference would only work with ideal agents, and even then it could only tell you what they want most among the possible options.