As I told someone else, this pdf has preliminary discussion about how to resolve differences that persist under extrapolation.
The specific example of religious disagreements seems like a trivial problem to anyone who gets far enough to consider the question. Since there aren’t any gods, the AI can ask what religious people would want if they accepted this fact. (This is roughly why I would oppose extrapolating only LW-ers rather than humanity as a whole.) But hey, maybe the question is more difficult than I think—we wouldn’t specifically tell the AI to be an atheist if general rules of thinking did not suffice—or maybe this focus on surface claims hides some deeper disagreement that can’t be so easily settled by probability.
As I told someone else, this pdf has preliminary discussion about how to resolve differences that persist under extrapolation.
The specific example of religious disagreements seems like a trivial problem to anyone who gets far enough to consider the question. Since there aren’t any gods, the AI can ask what religious people would want if they accepted this fact. (This is roughly why I would oppose extrapolating only LW-ers rather than humanity as a whole.) But hey, maybe the question is more difficult than I think—we wouldn’t specifically tell the AI to be an atheist if general rules of thinking did not suffice—or maybe this focus on surface claims hides some deeper disagreement that can’t be so easily settled by probability.