an essential truth is lost: while parts of CEV are left unspecified, other parts are, and so the output is not fully arbitrary.
What parts are specified? If the set of people is unspecified, the extrapolation procedure is unspecified, and the reconciliation procedure is unspecified, then what is left?
The set of CEV-compatible value systems is smaller than the set of possible value systems;
No. For all value systems X who are held by some people, you could always apply the CEV to a set of people who hold X. Unless the extrapolation procedure does something funny, the CEV of that set of people would be X.
On a practical level, it would mean that launching an AGI first becomes competitive, potentially replacing careful deliberation with a race to finish. And the risks of mistakes in a race to finish seem to far outweigh the importance of almost any slight differences in value systems.
Unless the extrapolation and the reconciliation procedures are trivial, computing the CEV of mankind would be probably beyond the possibility of any physically plausible AGI, superintelligent or not.
People here seem to assume AGI = omniscient deity, but there are no compelling technical reasons for that assumption. Most likely that’s just a reflection of traditional religious beliefs.
What parts are specified? If the set of people is unspecified, the extrapolation procedure is unspecified, and the reconciliation procedure is unspecified, then what is left?
No. For all value systems X who are held by some people, you could always apply the CEV to a set of people who hold X. Unless the extrapolation procedure does something funny, the CEV of that set of people would be X.
Unless the extrapolation and the reconciliation procedures are trivial, computing the CEV of mankind would be probably beyond the possibility of any physically plausible AGI, superintelligent or not.
People here seem to assume AGI = omniscient deity, but there are no compelling technical reasons for that assumption. Most likely that’s just a reflection of traditional religious beliefs.