If so, why would you believe it’s better to take an action that results in you having one billion and one utilons one-one-billionth of the time, and nothing all other times, than an action that reliably gives you one utilon?
One possible response is that the former action is preferable, but the intuition pump yields a different result because our intuitions are informed by actual small and large rewards (e.g., money), and in the real world getting $1 every day for eight years with certainty does not have the same utility as getting $2922 with probability 1/2922 each day for the next eight years. If real-world examples like money—which is almost always more valuable now than later, inflation aside; and which bears hidden and nonlinearly changing utilities like ‘security’ and ‘versatility’ and ‘social status’ and ‘peace of mind’ that we learn to reason with intuitively as though they could not be quantified in a single utility metric analogous to the currency measure itself—are the only intuitive grasp we have on ‘utilons,’ then we may make systematic errors in trying to cash out how our values would, if we better understood our biases, be reflectively cashed out.
One possible response is that the former action is preferable, but the intuition pump yields a different result because our intuitions are informed by actual small and large rewards (e.g., money), and in the real world getting $1 every day for eight years with certainty does not have the same utility as getting $2922 with probability 1/2922 each day for the next eight years. If real-world examples like money—which is almost always more valuable now than later, inflation aside; and which bears hidden and nonlinearly changing utilities like ‘security’ and ‘versatility’ and ‘social status’ and ‘peace of mind’ that we learn to reason with intuitively as though they could not be quantified in a single utility metric analogous to the currency measure itself—are the only intuitive grasp we have on ‘utilons,’ then we may make systematic errors in trying to cash out how our values would, if we better understood our biases, be reflectively cashed out.