(A) For all I, there exists some M such that I should observe M
and not
(B) There exists some set of moral facts M, such that for each intelligence I, I should observe M
right?
Eliezer uses “should” in an idiosyncratic way, which he thought (and maybe still thinks) would prevent a particular kind of confusion.
On this usage of “should”, Eliezer would probably* endorse something very close to (B). However, the “should” is with respect to the moral values towards which human CEV points (in the actual world, not in some counterfactual or future world in which the human CEV is different). These values make up the M that is asserted to exist in (B). And, as far as M is concerned, it would probably be best if all intelligent agents observed M.
* I’m hedging a little bit because maybe, under some perverse circumstances, it would be moral for an agent to be unmoved by moral facts. To give a fictional example, apparently God was in such a circumstance when he hardened the heart of Pharaoh.
Eliezer uses “should” in an idiosyncratic way, which he thought (and maybe still thinks) would prevent a particular kind of confusion.
On this usage of “should”, Eliezer would probably* endorse something very close to (B). However, the “should” is with respect to the moral values towards which human CEV points (in the actual world, not in some counterfactual or future world in which the human CEV is different). These values make up the M that is asserted to exist in (B). And, as far as M is concerned, it would probably be best if all intelligent agents observed M.
* I’m hedging a little bit because maybe, under some perverse circumstances, it would be moral for an agent to be unmoved by moral facts. To give a fictional example, apparently God was in such a circumstance when he hardened the heart of Pharaoh.