Instrumental rationality is fairly easy to define when it’s about optimising your own UF. You seem to have the worry that optimising your own UF isn’t what you should be doing given the default meaning of “should”. Which is fair enough.
In a naive conception, a lot of people have a utility function that values the pleasure of eating highly sugared food and not eating a lot of green vegetables.
Self-modifying in a way that you get more pleasure from eating more healthy food seems naively useful but it’s changing the utility function. It’s unclear to what extent it makes sense to do things that do shift one’s utility function.
Every new experience may change your preferences. Seeking out new experiences will predictably have this effect. Openness to experience requires openness to seeing your preferences changing, even if you do not know in what direction. This has been going on since birth, as no-one is born with the preferences they will have as adults. There is a process by which they develop, which does not, or at least need not, stop on reaching adulthood.
Yes, experiences change preferences. This process is however not just random but can be guided. The question of how that process of changing the utility function can be addressed rationally is open.
What utility function? There isn’t one answer, and I’m not certain it really is a function since different values within it may be incommensurable. It isn’t the easy question, it’s the impossible one. More concretely, what utility function should you use? Yours. Don’t know what it is? Find out. The necessary introspection is actual incredibly difficult, error prone, and full of special cases, so, good luck. Personally, I’d look into the values and virtues you truly hold dear, rather than utilities per se. (That does make the math not as useful, so you’ll have to train your intuition for it.)
Is suffering bad? According to most people, including me, yes, but it is hardly the most important thing. Many people’s utility function (still including me) says people should be willing to undergo extreme torture to save the people they truly love from lesser, but still bad, torture.
Is pleasure good? Sure, I prefer it, as do most people, but I’m fully anti-wireheading.
I see suffering and pleasure as simple, but useful proxies for the actual goodness of the state of the world according to that agent. Like any proxy, focusing too hard on the proxy itself (Goodharting) causes the proxy to no longer be useful as it stops representing that which it is a proxy of.
What is good? What is bad? Is suffering bad? Is pleasure good?
Rationality works well when there is a clear cut utility function, but what utility function is the good one?
Instrumental rationality is fairly easy to define when it’s about optimising your own UF. You seem to have the worry that optimising your own UF isn’t what you should be doing given the default meaning of “should”. Which is fair enough.
In a naive conception, a lot of people have a utility function that values the pleasure of eating highly sugared food and not eating a lot of green vegetables.
Self-modifying in a way that you get more pleasure from eating more healthy food seems naively useful but it’s changing the utility function. It’s unclear to what extent it makes sense to do things that do shift one’s utility function.
“I don’t like spinach, and I’m glad I don’t, because if I liked it i’d eat it, and I hate it.”
Every new experience may change your preferences. Seeking out new experiences will predictably have this effect. Openness to experience requires openness to seeing your preferences changing, even if you do not know in what direction. This has been going on since birth, as no-one is born with the preferences they will have as adults. There is a process by which they develop, which does not, or at least need not, stop on reaching adulthood.
Yes, experiences change preferences. This process is however not just random but can be guided. The question of how that process of changing the utility function can be addressed rationally is open.
What utility function? There isn’t one answer, and I’m not certain it really is a function since different values within it may be incommensurable. It isn’t the easy question, it’s the impossible one. More concretely, what utility function should you use? Yours. Don’t know what it is? Find out. The necessary introspection is actual incredibly difficult, error prone, and full of special cases, so, good luck. Personally, I’d look into the values and virtues you truly hold dear, rather than utilities per se. (That does make the math not as useful, so you’ll have to train your intuition for it.)
Is suffering bad? According to most people, including me, yes, but it is hardly the most important thing. Many people’s utility function (still including me) says people should be willing to undergo extreme torture to save the people they truly love from lesser, but still bad, torture.
Is pleasure good? Sure, I prefer it, as do most people, but I’m fully anti-wireheading.
I see suffering and pleasure as simple, but useful proxies for the actual goodness of the state of the world according to that agent. Like any proxy, focusing too hard on the proxy itself (Goodharting) causes the proxy to no longer be useful as it stops representing that which it is a proxy of.