Because human utility functions are not just composed of “the beautiful and holy”. It would be hard to even disentangle the beautiful and holy from the primitive and selfish parts, since after all, they came from the same place, not like they’re fundamentally different. There is something unnatural, disturbing, abstractly distasteful to me about a being which artificially has its utility function maximized through powerful optimization, even if that being is myself. I gave the example of wireheading an ant.
Because human utility functions are not just composed of “the beautiful and holy”. It would be hard to even disentangle the beautiful and holy from the primitive and selfish parts, since after all, they came from the same place, not like they’re fundamentally different. There is something unnatural, disturbing, abstractly distasteful to me about a being which artificially has its utility function maximized through powerful optimization, even if that being is myself. I gave the example of wireheading an ant.