Without a narrow enough prior on the space of possible preferences,
But we’ve already developed complex models of what ‘intelligent’ beings ‘should’ prefer—based on our own preferences. Try to explore beyond that intuitive model, and we become very uncomfortable.
When speculating about real intelligences, alien or artificial, it becomes very important that we justify the restrictions we place on the set of potential preferences we consider they might have.
But we’ve already developed complex models of what ‘intelligent’ beings ‘should’ prefer—based on our own preferences. Try to explore beyond that intuitive model, and we become very uncomfortable.
When speculating about real intelligences, alien or artificial, it becomes very important that we justify the restrictions we place on the set of potential preferences we consider they might have.