This seems on the right track, but maybe not pushed far enough. I think all human abilities (instincts, math, philosophy, etc) arose in a specific kind of environment and can’t be uniquely extended to all possible environments in our universe, never mind simulated ones. I don’t trust any extrapolation procedures carried very far, because they seem too sensitive (our thinking about weird situations is already quite divergent). So I guess FAI will need to give us an environment that’s good and mostly normal. That way it can skip rescuing many of our preferences, because in normal environments they work fine anyway.
This seems on the right track, but maybe not pushed far enough. I think all human abilities (instincts, math, philosophy, etc) arose in a specific kind of environment and can’t be uniquely extended to all possible environments in our universe, never mind simulated ones. I don’t trust any extrapolation procedures carried very far, because they seem too sensitive (our thinking about weird situations is already quite divergent). So I guess FAI will need to give us an environment that’s good and mostly normal. That way it can skip rescuing many of our preferences, because in normal environments they work fine anyway.