A hypothetical idealization dynamic may not be helpful in actually making decisions, but its theoretical role is that it provides a possible specification of the “territory” that moral reasoning should explore, a criterion of correctness.
I think that such a specification probably isn’t the correct specification of the territory that moral reasoning should explore. By analogy, it’s like specifying the territory for mathematical reasoning based on idealizing human mathematical reasoning, or specifying the territory for scientific reasoning based on idealizing human scientific reasoning. (As opposed to figuring out how to directly refer to some external reality.) It seems like a step that’s generally tempting to take when you’re able to informally reason (to some extent) about something but you don’t know how to specify the territory, but I would prefer to just say that we don’t know how to specify the territory yet. But...
It is a hard-to-use criterion of correctness, you might need to build a FAI to actually access it, but at least it’s meaningful, and it illustrates the way in which many ways of thinking about morality are confused.
Maybe I’m underestimating the utility of having a specification that’s “at least meaningful” even if it’s not necessarily correct. (I don’t mind “hard-to-use” so much.) Can you give some examples of how it illustrates the way in which many ways of thinking about morality are confused?
I think that such a specification probably isn’t the correct specification of the territory that moral reasoning should explore. By analogy, it’s like specifying the territory for mathematical reasoning based on idealizing human mathematical reasoning, or specifying the territory for scientific reasoning based on idealizing human scientific reasoning. (As opposed to figuring out how to directly refer to some external reality.) It seems like a step that’s generally tempting to take when you’re able to informally reason (to some extent) about something but you don’t know how to specify the territory, but I would prefer to just say that we don’t know how to specify the territory yet. But...
Maybe I’m underestimating the utility of having a specification that’s “at least meaningful” even if it’s not necessarily correct. (I don’t mind “hard-to-use” so much.) Can you give some examples of how it illustrates the way in which many ways of thinking about morality are confused?