These are old questions, and certainly they’ll need to be answered. There is no running away from having a good theory that explains why the take-off process can be trusted to do the right thing. Ideally, this theory will be as clear-cut as classical mechanics.
The data is in the process of humanity, but so is the right way of interpreting it. How much do you need to specify by hand to launch the process of extracting preference of humanity formally, that knows what to look for, and is able to find enough to continue looking for more? It’d take at least a technical theory of preference generalized to non-formal agents (of which the humanity is an instance) to answer that question.
These are old questions, and certainly they’ll need to be answered. There is no running away from having a good theory that explains why the take-off process can be trusted to do the right thing. Ideally, this theory will be as clear-cut as classical mechanics.
The data is in the process of humanity, but so is the right way of interpreting it. How much do you need to specify by hand to launch the process of extracting preference of humanity formally, that knows what to look for, and is able to find enough to continue looking for more? It’d take at least a technical theory of preference generalized to non-formal agents (of which the humanity is an instance) to answer that question.