I think Nate Soares has beliefs about question 1. A few weeks ago, we were discussing a question that seems analogous to me—“does moral deliberation converge, for different ways of doing moral deliberation? E.g. is there a unique human CEV?”—and he said he believes the answer is “yes.” I didn’t get the chance to ask him why, though.
Thinking about it myself for a few minutes, it does feel like all of your examples for how the overseer could have distorted values have a true “wrongness” about them that can be verified against reality—this makes me feel optimistic that there is a basin of human values, and that “interacting with reality” broadly construed is what draws you in.
I think Nate Soares has beliefs about question 1. A few weeks ago, we were discussing a question that seems analogous to me—“does moral deliberation converge, for different ways of doing moral deliberation? E.g. is there a unique human CEV?”—and he said he believes the answer is “yes.” I didn’t get the chance to ask him why, though.
Thinking about it myself for a few minutes, it does feel like all of your examples for how the overseer could have distorted values have a true “wrongness” about them that can be verified against reality—this makes me feel optimistic that there is a basin of human values, and that “interacting with reality” broadly construed is what draws you in.