I’m very confused by the notion of “not having a utility function”. My understanding of utility function is that it’s impossible not to have one, even if the function is implicit, subconscious, or something that wouldn’t be endorsed if it could be stated explicitly.
It seems like when you’re saying the CDC chair doesn’t have a utility function, you mean something like “the politics term in the utility function dominates all other terms”. But perhaps I’ve misunderstood you, or I misunderstand the meaning of “utility function” in this context.
My understanding of utility function is that it’s impossible not to have one, even if the function is implicit, subconscious, or something that wouldn’t be endorsed if it could be stated explicitly.
My understanding is that a utility function implies consistent and coherent preferences. Humans definitely don’t have that, our preferences are inconsistent and subject to your instance framing effects.
Thats the correct definition, but rationalists have got into the habit of using “utlity function” to mean prefefrences, leading to considerable confusion.
I’ve been interpreting ‘utility function’ along the lines of ‘coherent extrapolated volition’, i.e. something like ‘the most similar utility function’ that’s both coherent and consistent and best approximates ‘preferences’.
The intuition is that there is, in some sense, an adjacent or nearby utility function, even if human behavior isn’t (perfectly) consistent or coherent.
“Behavior-executor”: acts on reflex, producing a fixed action in response to a fixed stimulus (regardless of how this corresponds to outcomes).
“Utility-maximizer”: chooses actions based on their expected outcomes; makes long-term plans, and completely changes behavior if new information comes in suggesting their old behavior patterns aren’t helping produce the desired outcomes.
I’m very confused by the notion of “not having a utility function”. My understanding of utility function is that it’s impossible not to have one, even if the function is implicit, subconscious, or something that wouldn’t be endorsed if it could be stated explicitly.
It seems like when you’re saying the CDC chair doesn’t have a utility function, you mean something like “the politics term in the utility function dominates all other terms”. But perhaps I’ve misunderstood you, or I misunderstand the meaning of “utility function” in this context.
My understanding is that a utility function implies consistent and coherent preferences. Humans definitely don’t have that, our preferences are inconsistent and subject to your instance framing effects.
Thats the correct definition, but rationalists have got into the habit of using “utlity function” to mean prefefrences, leading to considerable confusion.
I’ve been interpreting ‘utility function’ along the lines of ‘coherent extrapolated volition’, i.e. something like ‘the most similar utility function’ that’s both coherent and consistent and best approximates ‘preferences’.
The intuition is that there is, in some sense, an adjacent or nearby utility function, even if human behavior isn’t (perfectly) consistent or coherent.
I think Zvi is drawing on this informal distinction in The Blue-Minimizing Robot:
“Behavior-executor”: acts on reflex, producing a fixed action in response to a fixed stimulus (regardless of how this corresponds to outcomes).
“Utility-maximizer”: chooses actions based on their expected outcomes; makes long-term plans, and completely changes behavior if new information comes in suggesting their old behavior patterns aren’t helping produce the desired outcomes.
I was also imagining the distinctions of
adaptation-executers vs. fitness-maximizers
and
selection + unconscious reinforcement vs. conscious strategizing
which are similar.
Thanks, this (and the sister comment by Unnamed) makes perfect sense.