An example of the other way to approach this question is the idea of simulating a group of consciousness theorists for 500 subjective years, until they arrive at a consensus on the nature of consciousness. I think it’s rather unlikely that anyone will ever get to solve FAI-relevant problems in that way.
The CEV idea there would be to create an AI which is optimizing for expected satisfaction of the utility function that would be output by such a process. If the AI’s other functionality is good, it will start with reasonable guesses about what such a process would output, and rapidly improve those guesses. As it further improved, gathered more data, etc, it would better and better approximate that output.
The CEV idea there would be to create an AI which is optimizing for expected satisfaction of the utility function that would be output by such a process. If the AI’s other functionality is good, it will start with reasonable guesses about what such a process would output, and rapidly improve those guesses. As it further improved, gathered more data, etc, it would better and better approximate that output.