Because human utility functions are not just composed of “the beautiful and holy”. It would be hard to even disentangle the beautiful and holy from the primitive and selfish parts, since after all, they came from the same place, not like they’re fundamentally different. There is something unnatural, disturbing, abstractly distasteful to me about a being which artificially has its utility function maximized through powerful optimization, even if that being is myself. I gave the example of wireheading an ant.
pantalaimon
Karma: −3
How can rationalists perform exceptionally well?
disgust at utility maximization
pantalaimon’s Shortform
Attempting to revitalize the rationalist community on discord. Join us: https://discord.gg/d5fdXheJ3t
Why am I being downvoted for asking a question?
It’s an important question to me since it can feel like you’re completely blind on important and complex goals in life, unable to know for sure whether any of your actions are even helping.
And obviously we want to have more instrumental rationality...