The principal challenge of CEV is not the part where you take into account all the specific, contingent individuals who exist on the earth, in the course of following some recipe for extrapolating, aggregating, etc. The important step is figuring out the “recipe” itself. If we had a recipe which we knew to perfectly reflect true human ideals, but it proved to be computationally infeasible, we (or a FAI) could then think about feasible approximations to this ideal.
The important questions are: What sort of decision system is a typical human being? What are the values and value templates with which it comes equipped? These questions should have exact answers just as objective as the facts about how many limbs or how many bones a typical human has, and those answers should in turn imply that “friendliness” and “CEV” ought to work in some specific way.
The principal challenge of CEV is not the part where you take into account all the specific, contingent individuals who exist on the earth, in the course of following some recipe for extrapolating, aggregating, etc. The important step is figuring out the “recipe” itself. If we had a recipe which we knew to perfectly reflect true human ideals, but it proved to be computationally infeasible, we (or a FAI) could then think about feasible approximations to this ideal.
The important questions are: What sort of decision system is a typical human being? What are the values and value templates with which it comes equipped? These questions should have exact answers just as objective as the facts about how many limbs or how many bones a typical human has, and those answers should in turn imply that “friendliness” and “CEV” ought to work in some specific way.