Here’s another objection. Putting someone in an environment they control completely which has unlimited computational power could lead to some pretty unexpected stuff. Wireheading would be easy, and it could start innocuously: I decide I could use an attractive member of my preferred gender to keep me company and things get worse from there. If you put someone in this situation it seems like there’d be tremendous incentives to procrastinate indefinitely on solving the problem at hand.
It seems like under ideal conditions we could empirically test the behavior of this sort of exotic “utility function” and make sure it was meeting basic sanity checks.
Here’s another objection. Putting someone in an environment they control completely which has unlimited computational power could lead to some pretty unexpected stuff. Wireheading would be easy, and it could start innocuously: I decide I could use an attractive member of my preferred gender to keep me company and things get worse from there. If you put someone in this situation it seems like there’d be tremendous incentives to procrastinate indefinitely on solving the problem at hand.
It seems like under ideal conditions we could empirically test the behavior of this sort of exotic “utility function” and make sure it was meeting basic sanity checks.