I like what you’re trying to do and I hope you succeed, but I have some nitpicks.
Firstly, what the utility function is and what it’s not. It belongs to the field of instrumental rationality, not epistemic rationality; it is not part of the territory. Don’t expect it to correspond to something physical.
My utility function corresponds to specific structures in my brain.
That’s it—you now have a shiny new utility function. And I want to help you optimize it. (Though it can grow and develop and change along with yourself; I want this to be a speculative process, not one in which you suddenly commit to an immutable life goal).
I don’t think that it’s that easy to figure out a utility function. There have been many things in my life that I’ve thought about and decided weren’t that important, but then when I’ve actually tried them I discovered that I really really enjoy them. (Like, having a girlfriend, large numbers of friends, social hobbies, etc.)
You could say that that’s a failure of my epistemic rationality, but I suspect that signalling is going to dominate hypotheticals that I think about but have never actually experienced. I think I would need a lot of practice in the inside view to be able to actually judge if I want something without trying it first.
My utility function corresponds to specific structures in my brain.
Do you mean there’s a particular structure in your brain sending out “good” and “bad” messages, and that’s what you’re trying to optimize? (i.e. that you would choose to wirehead).
Or do you mean in the sense that “utility functions have something to do with expressed and revealed preferences which necessarily correspond to structures in my brain?”
here have been many things in my life that I’ve thought about and decided weren’t that important
Could they be covered under the umbrella of “I value things which I would enjoy if I tried”?
Or do you mean in the sense that “utility functions have something to do with expressed and revealed preferences which necessarily correspond to structures in my brain?”
More the latter.
Could they be covered under the umbrella of “I value things which I would enjoy if I tried”?
Yes, but I don’t know how to optimize for those things (other than being generally adventurous/curious), since I don’t know if I would enjoy them.
I like what you’re trying to do and I hope you succeed, but I have some nitpicks.
My utility function corresponds to specific structures in my brain.
I don’t think that it’s that easy to figure out a utility function. There have been many things in my life that I’ve thought about and decided weren’t that important, but then when I’ve actually tried them I discovered that I really really enjoy them. (Like, having a girlfriend, large numbers of friends, social hobbies, etc.)
You could say that that’s a failure of my epistemic rationality, but I suspect that signalling is going to dominate hypotheticals that I think about but have never actually experienced. I think I would need a lot of practice in the inside view to be able to actually judge if I want something without trying it first.
Do you mean there’s a particular structure in your brain sending out “good” and “bad” messages, and that’s what you’re trying to optimize? (i.e. that you would choose to wirehead).
Or do you mean in the sense that “utility functions have something to do with expressed and revealed preferences which necessarily correspond to structures in my brain?”
Could they be covered under the umbrella of “I value things which I would enjoy if I tried”?
More the latter.
Yes, but I don’t know how to optimize for those things (other than being generally adventurous/curious), since I don’t know if I would enjoy them.