Because human utility functions are not just composed of “the beautiful and holy”. It would be hard to even disentangle the beautiful and holy from the primitive and selfish parts, since after all, they came from the same place, not like they’re fundamentally different. There is something unnatural, disturbing, abstractly distasteful to me about a being which artificially has its utility function maximized through powerful optimization, even if that being is myself. I gave the example of wireheading an ant.
Hmm. I’m not sure what you’re disgusted by. If the complaint is that humans (all/most?) aren’t pure, and have motives that seem gross and base, you’re probably right, but also probably not well-served by being disgusted—finding beauty in imperfection or just curiosity about what makes us tick might be better. That said, I’m not one to yuck your … yuck.
If the complaint is that utility theory is itself gross because it allows it, I don’t agree. It’s still a useful model, just that reality clashes with your preferences.
Because human utility functions are not just composed of “the beautiful and holy”. It would be hard to even disentangle the beautiful and holy from the primitive and selfish parts, since after all, they came from the same place, not like they’re fundamentally different. There is something unnatural, disturbing, abstractly distasteful to me about a being which artificially has its utility function maximized through powerful optimization, even if that being is myself. I gave the example of wireheading an ant.
Hmm. I’m not sure what you’re disgusted by. If the complaint is that humans (all/most?) aren’t pure, and have motives that seem gross and base, you’re probably right, but also probably not well-served by being disgusted—finding beauty in imperfection or just curiosity about what makes us tick might be better. That said, I’m not one to yuck your … yuck.
If the complaint is that utility theory is itself gross because it allows it, I don’t agree. It’s still a useful model, just that reality clashes with your preferences.