If you’re considering the ability to rewire one’s utility function, why simplify the function rather than build cognitive tools to help people better satisfy the function? What you’ve proposed is that an AI destroys human intelligence, then pursues some abstraction of what it thinks humans wanted.
Your suggestion is that an AI might assume that the best way to reach its goal of making humans happy (maximizing their utility) is to attain the ends of humans’ functions faster and better than we could, and rewire us to be satisfied. There are two problems here that I see.
First, the means are an end. Much of what we value isn’t the goal we claim as our objective, but the process of striving for the goal. So here you have an AI that doesn’t really understand what humans want.
Second, most humans aren’t interested in every avenue of exploration, creation and pleasure. Our interests are distinct. They also do change over time (or some limited set of parameters do anyhow). We don’t always notice them change, and when they do, we like to track down what decisions they made that led them to their new preferences. People value the (usually illusory) notion that they control changes to their utility functions. The offer to be a “wirehead” is an action which intrinsically violates peoples’ utility in the illusion of autonomy. This doesn’t apply to everyone—hedonists can apply to your heaven. I suspect that few others would want it.
Also,
If you don’t want to be “reduced” to an eternal state of bliss, that’s tough luck… The FAI can simply modify your preferences so you want an eternally blissful state.
That is not friendly.
I think you have an idea that there is a “global human utility function” and that FAI is that which satisfies this. Humans have commonalities in their functions, but they are localized around the notion of self. Your “FAI” generalizes what most people want in some form except for experience and autonomy, but the other values it extracts are, in humans, not independent from those.
If you’re considering the ability to rewire one’s utility function, why simplify the function rather than build cognitive tools to help people better satisfy the function? What you’ve proposed is that an AI destroys human intelligence, then pursues some abstraction of what it thinks humans wanted.
Your suggestion is that an AI might assume that the best way to reach its goal of making humans happy (maximizing their utility) is to attain the ends of humans’ functions faster and better than we could, and rewire us to be satisfied. There are two problems here that I see.
First, the means are an end. Much of what we value isn’t the goal we claim as our objective, but the process of striving for the goal. So here you have an AI that doesn’t really understand what humans want.
Second, most humans aren’t interested in every avenue of exploration, creation and pleasure. Our interests are distinct. They also do change over time (or some limited set of parameters do anyhow). We don’t always notice them change, and when they do, we like to track down what decisions they made that led them to their new preferences. People value the (usually illusory) notion that they control changes to their utility functions. The offer to be a “wirehead” is an action which intrinsically violates peoples’ utility in the illusion of autonomy. This doesn’t apply to everyone—hedonists can apply to your heaven. I suspect that few others would want it.
Also,
That is not friendly.
I think you have an idea that there is a “global human utility function” and that FAI is that which satisfies this. Humans have commonalities in their functions, but they are localized around the notion of self. Your “FAI” generalizes what most people want in some form except for experience and autonomy, but the other values it extracts are, in humans, not independent from those.