The resemblance between my second suggestion and your thing didn’t go unnoticed—I had in fact read your coherent extrapolated volition thing before (there’s probably an old e-mail from me to you about it, in fact). I think it’s basically correct. But the method of justification is importantly different, because the idea is that we’re trying to approximate something with epistemic content—we’re not just trying to do what you might call a Xannon thing—we’re not just trying to model what humans would do. Rather, we’re trying to model and improve a specific feature of humanity that we see as morally relevant—responsiveness to reasons.
That’s really, really important.
In the context of your dialogue above, it’s what reconciles Xannon and Yancy: even if Yancy can’t convince Xannon that there’s some kind of non-subjective moral truth, he ought to be able to convince Xannon that moral beliefs should be responsive to reasons—and likewise, even if Xannon can’t convince Yancy that what really matters, morally, is what people can agree on, he should be able to convince Yancy that the best way to get at it in the real world is by a collective process of reasoning.
So you see that this method of justification does provide a way to answers to questions like “friendliness to whom.” I know what I’m doing, Eliezer. :-)
Eliezer,
The resemblance between my second suggestion and your thing didn’t go unnoticed—I had in fact read your coherent extrapolated volition thing before (there’s probably an old e-mail from me to you about it, in fact). I think it’s basically correct. But the method of justification is importantly different, because the idea is that we’re trying to approximate something with epistemic content—we’re not just trying to do what you might call a Xannon thing—we’re not just trying to model what humans would do. Rather, we’re trying to model and improve a specific feature of humanity that we see as morally relevant—responsiveness to reasons.
That’s really, really important.
In the context of your dialogue above, it’s what reconciles Xannon and Yancy: even if Yancy can’t convince Xannon that there’s some kind of non-subjective moral truth, he ought to be able to convince Xannon that moral beliefs should be responsive to reasons—and likewise, even if Xannon can’t convince Yancy that what really matters, morally, is what people can agree on, he should be able to convince Yancy that the best way to get at it in the real world is by a collective process of reasoning.
So you see that this method of justification does provide a way to answers to questions like “friendliness to whom.” I know what I’m doing, Eliezer. :-)