In the present world we are forced, due to resource constraints, between virtuous values and hedonistic values. It isn’t that we want to force ourselves to choose the virtuous, we want to dissolve the need to choose between them so we can satisfy both.
Not sure I completely understand, but is this a possibility under your proposal?
AI: “Voila!”
Humans: “Uh, that’s not what we want.”
AI: “No, but it’s what you want to want.”
Hopefully the AI makes us want what we want to want. Then we’d say…
Humans: “Yay! Broccoli for dinner again!”
In the present world we are forced, due to resource constraints, between virtuous values and hedonistic values. It isn’t that we want to force ourselves to choose the virtuous, we want to dissolve the need to choose between them so we can satisfy both.