I meant: innate cognitive architecture which plays a role in metaethical thought.
You might be familiar with the idea that, according to CEV, you figure out the full complexity of human value using neuroscience (rather than relying on people’s opinions about what they value), and then you “extrapolate” or “renormalize” that using “reflective decision theory” (which does not yet exist). The idea here is that the method of extrapolation should also be extracted from the details of human cognitive architecture, rather than just figured out through intuition or pure reason.
Suppose we have a person—or an intelligent agent—with a particular “value system” or “private decision theory”. Note that we are talking about its actual decision theory, as embodied in its causal structure and decision-making dispositions, and not just its introspective opinions about how it decides. Given this actual value system, RDT is supposed to tell us what would happen to that value system if it were changed according to its own implicit ideals. All I’m saying is that there’s a meta-ethical relativism for RDT, for large classes of decision architecture. Different theories about how to normatively self-modify a decision architecture ought to be possible, and the selection of which RDT is used should also be derived from the agent’s own cognitive architecture.
Of course you can go meta again and say, maybe the RDT extraction procedure can also take different forms—etc. It’s one of the tasks of the FAI/CEV/RDT research program to figure out when and how the ethical metalevels stop.
I meant: innate cognitive architecture which plays a role in metaethical thought.
You might be familiar with the idea that, according to CEV, you figure out the full complexity of human value using neuroscience (rather than relying on people’s opinions about what they value), and then you “extrapolate” or “renormalize” that using “reflective decision theory” (which does not yet exist). The idea here is that the method of extrapolation should also be extracted from the details of human cognitive architecture, rather than just figured out through intuition or pure reason.
Suppose we have a person—or an intelligent agent—with a particular “value system” or “private decision theory”. Note that we are talking about its actual decision theory, as embodied in its causal structure and decision-making dispositions, and not just its introspective opinions about how it decides. Given this actual value system, RDT is supposed to tell us what would happen to that value system if it were changed according to its own implicit ideals. All I’m saying is that there’s a meta-ethical relativism for RDT, for large classes of decision architecture. Different theories about how to normatively self-modify a decision architecture ought to be possible, and the selection of which RDT is used should also be derived from the agent’s own cognitive architecture.
Of course you can go meta again and say, maybe the RDT extraction procedure can also take different forms—etc. It’s one of the tasks of the FAI/CEV/RDT research program to figure out when and how the ethical metalevels stop.