I don’t agree it’s true that we have a coherent set of preferences for each environment.
I’m sure we can agree that humans don’t have their utility function written down in FORTRAN on the inside of our skulls. Nor does our brain store a real number associated with each possible state of the universe (and even if we did, by what lights would we call that number a utility function?).
So when we talk about a human’s preferences in some environment, we’re not talking about opening them up and looking at their brain, we’re talking how humans have this propensity to take reasonable actions that make sense in terms of preferences. Example: You say “would you like doritos or an apple?” and I say “apple,” and then you use this behavior to update your model of my preferences.
But this action-propensity that humans have is sometimes irrational (bold claim I know) and not so easily modeled as a utility function, even within a single environment.
The scheme you talk about for building up human values seems to have a recursive character to it: you get the bigger, broader human utility function by building it out of smaller, more local human utility functions, and so on, until at some base level of recursion there are utility functions that get directly inferred from facts about the human. But unless there’s some level of human action where we act like rational utility maximizers, this base level already contains the problems I’m talking about, and since it’s the base level those problems can’t be resolved or explained by recourse to a yet-baser level.
Different people have different responses to this problem, and I think it’s legitimate to say “well, just get better at inferring utility functions” (though this requires some actual work at specifying a “better”). But I’m going to end up arguing that we should just get better at dealing with models of preferences that aren’t utility functions.
Thanks for the comment :)
I don’t agree it’s true that we have a coherent set of preferences for each environment.
I’m sure we can agree that humans don’t have their utility function written down in FORTRAN on the inside of our skulls. Nor does our brain store a real number associated with each possible state of the universe (and even if we did, by what lights would we call that number a utility function?).
So when we talk about a human’s preferences in some environment, we’re not talking about opening them up and looking at their brain, we’re talking how humans have this propensity to take reasonable actions that make sense in terms of preferences. Example: You say “would you like doritos or an apple?” and I say “apple,” and then you use this behavior to update your model of my preferences.
But this action-propensity that humans have is sometimes irrational (bold claim I know) and not so easily modeled as a utility function, even within a single environment.
The scheme you talk about for building up human values seems to have a recursive character to it: you get the bigger, broader human utility function by building it out of smaller, more local human utility functions, and so on, until at some base level of recursion there are utility functions that get directly inferred from facts about the human. But unless there’s some level of human action where we act like rational utility maximizers, this base level already contains the problems I’m talking about, and since it’s the base level those problems can’t be resolved or explained by recourse to a yet-baser level.
Different people have different responses to this problem, and I think it’s legitimate to say “well, just get better at inferring utility functions” (though this requires some actual work at specifying a “better”). But I’m going to end up arguing that we should just get better at dealing with models of preferences that aren’t utility functions.