What I am trying to fathom is the difference between 1.) assigning utility arbitrarily (no objective grounding) 2.) grounding utility in units of bodily sensations 3.) grounding utility in units of human well-being (i.e. number of conscious beings whose life’s are worth living).
Retains complex values but might be inconsistent (e.g., how do you assign utility to novel goals that can’t be judged in terms of previous goals (e.g., hunter gatherer who learns about category theory)).
You assign most utility to universes that maximize desirable bodily sensations (likely effect: wireheading (reduction of complex values to a narrow set of values (e.g., the well-being of other humans is abandoned because bodily sensations do not grow linearly with the number of beings that are saved))).
Same as #2 except that complex values are now reduced to the well-being of other humans.
As you see, my problem is that to me as a complete layman expected utility maximization seems to lead to the reduction of complex values once it is measured in some objectively verifiable physical fact. In other words, as long as utility is dimensionless, it seems to be an inconsistent measure, if you add a dimension it leads to the destruction of complex values.
The downvoting of the OP seems to suggest that some people seem to suspect that I am not honest, but I am really interested to learn more about this and how I am wrong. I am not trying to claim some insight here but merely ask people for help who understand a lot more about it than me. I am also not selfish, as some people seem to think? I care strongly about other humans and even lower animals.
how do you assign utility to novel goals that can’t be judged in terms of previous goals
Don’t think in terms of choosing what value to assign, think in terms of figuring out what value your utility functions already assigns to it (your utility function is a mathematical object that always has and always will exist).
So the answer is that you can’t be expected to know yet what your value your utility function assigns to goals you haven’t thought of, and this doesn’t matter too much since uncertainty about your utility function can just be treated like any other uncertainty.
The downvoting of the OP seems to suggest that some people seem to suspect that I am not honest, but I am really interested to learn more about this and how I am wrong.
For the record, I voted the OP up, because it made me think and in particular made me realise my utility function wasn’t additive or even approximately additive, which I had been unsure of before.
Don’t think in terms of choosing what value to assign, think in terms of figuring out what value your utility functions already assigns to it...
I don’t think that is possible. Consider the difference between a hunter-gatherer, who cares about his hunting success and to become the new clan chief, and a member of lesswrong who wants to determine if a “sufficiently large randomized Conway board could turn out to converge to a barren ‘all off’ state.”
The utility of the success in hunting down animals and proving abstract conjectures about cellular automata is largely determined by factors such as your education, culture and environmental circumstances. The same hunter gatherer who cared to kill a lot of animals, to get the best ladies in its clan, might have under different circumstances turned out to be a vegetarian mathematicians solely caring about his understanding of the nature of reality. Both sets of values are to some extent mutually exclusive or at least disjoint. Yet both sets of values are what the person wants, given the circumstances. Change the circumstances dramatically and you change the persons values.
You might conclude that what the hunter-gatherer really wants is to solve abstract mathematical problems, he just doesn’t know about that. But there is no set of values that a person really wants. Humans are largely defined by the circumstances they reside in. If you already knew a movie, you wouldn’t watch it. To be able to get your meat from the supermarket changes the value of hunting.
If “we knew more, thought faster, were more the people we wished we were, and had grown up closer together” then we would stop to desire what we learnt, wish to think even faster, become even different people and get bored of and rise up from the people similar to us.
What I am trying to fathom is the difference between 1.) assigning utility arbitrarily (no objective grounding) 2.) grounding utility in units of bodily sensations 3.) grounding utility in units of human well-being (i.e. number of conscious beings whose life’s are worth living).
Retains complex values but might be inconsistent (e.g., how do you assign utility to novel goals that can’t be judged in terms of previous goals (e.g., hunter gatherer who learns about category theory)).
You assign most utility to universes that maximize desirable bodily sensations (likely effect: wireheading (reduction of complex values to a narrow set of values (e.g., the well-being of other humans is abandoned because bodily sensations do not grow linearly with the number of beings that are saved))).
Same as #2 except that complex values are now reduced to the well-being of other humans.
As you see, my problem is that to me as a complete layman expected utility maximization seems to lead to the reduction of complex values once it is measured in some objectively verifiable physical fact. In other words, as long as utility is dimensionless, it seems to be an inconsistent measure, if you add a dimension it leads to the destruction of complex values.
The downvoting of the OP seems to suggest that some people seem to suspect that I am not honest, but I am really interested to learn more about this and how I am wrong. I am not trying to claim some insight here but merely ask people for help who understand a lot more about it than me. I am also not selfish, as some people seem to think? I care strongly about other humans and even lower animals.
Don’t think in terms of choosing what value to assign, think in terms of figuring out what value your utility functions already assigns to it (your utility function is a mathematical object that always has and always will exist).
So the answer is that you can’t be expected to know yet what your value your utility function assigns to goals you haven’t thought of, and this doesn’t matter too much since uncertainty about your utility function can just be treated like any other uncertainty.
For the record, I voted the OP up, because it made me think and in particular made me realise my utility function wasn’t additive or even approximately additive, which I had been unsure of before.
I don’t think that is possible. Consider the difference between a hunter-gatherer, who cares about his hunting success and to become the new clan chief, and a member of lesswrong who wants to determine if a “sufficiently large randomized Conway board could turn out to converge to a barren ‘all off’ state.”
The utility of the success in hunting down animals and proving abstract conjectures about cellular automata is largely determined by factors such as your education, culture and environmental circumstances. The same hunter gatherer who cared to kill a lot of animals, to get the best ladies in its clan, might have under different circumstances turned out to be a vegetarian mathematicians solely caring about his understanding of the nature of reality. Both sets of values are to some extent mutually exclusive or at least disjoint. Yet both sets of values are what the person wants, given the circumstances. Change the circumstances dramatically and you change the persons values.
You might conclude that what the hunter-gatherer really wants is to solve abstract mathematical problems, he just doesn’t know about that. But there is no set of values that a person really wants. Humans are largely defined by the circumstances they reside in. If you already knew a movie, you wouldn’t watch it. To be able to get your meat from the supermarket changes the value of hunting.
If “we knew more, thought faster, were more the people we wished we were, and had grown up closer together” then we would stop to desire what we learnt, wish to think even faster, become even different people and get bored of and rise up from the people similar to us.