I don’t mean to imply that, given that we’ve performed a bottom-up extrapolation of human values into a coherent whole, that implementing that whole in a system vastly different from a human is necessarily difficult.
Indeed, by comparison to the first part, it’s almost undoubtedly trivial, as you suggest.
Rather, I mean that what is at issue is extrapolating the currently instantiated value systems into “a coherent whole that can be implemented by a system vastly different from a human”.
That said, I do think it’s worthwhile to distinguish between “Catholicism” and “the result of extrapolating Catholicism into a coherent whole.” The latter, supposing it existed, might not qualify as an example of the former. The same is true of “human value”.
I don’t mean to imply that, given that we’ve performed a bottom-up extrapolation of human values into a coherent whole, that implementing that whole in a system vastly different from a human is necessarily difficult.
Indeed, by comparison to the first part, it’s almost undoubtedly trivial, as you suggest.
Rather, I mean that what is at issue is extrapolating the currently instantiated value systems into “a coherent whole that can be implemented by a system vastly different from a human”.
That said, I do think it’s worthwhile to distinguish between “Catholicism” and “the result of extrapolating Catholicism into a coherent whole.” The latter, supposing it existed, might not qualify as an example of the former. The same is true of “human value”.