First of all, when I (Base Dorikka, BD) am using the phrases “X is moral,” “X is right,” and “I should X” in this comment, I am hypothesizing that a version of myself with a comparatively large amount of computing resources and information (Idealized Dorikka, ID) would come to the conclusion that action X would optimize ID’s utility function in relation to the set of all actions that BD could take in the situation.
Note that I am not stating with that what I say is “moral” would for certain optimize ID’s utility function in relation to the set of all actions that BD could take in the situation—my saying that “X is moral” represents me making a probability estimate that I would do so. It’s key to my understanding here that I don’t actually have any authority to declare something the moral thing, I am just estimating that the probability that X is moral is high.
This is my understanding of the use of such phrases—if you disagree with my conclusions, check if we disagree on how Eliezer uses the words.
Now, BD does not, of course, have access to ID’s computing resources, and so I can only imagine a version of ID a certain level above my own. However, ID would have enough computing resources to imagine an ID+ that was a certain level above him. If ID+ ever came to exist, he would be able to imagine an ID++, and so on.
I think that Eliezer’s metaethics would select option #2, and point out that ID’s reference for what he should do is ID+, not himself.
Edit: I, however, do have the objection that it isn’t meaningful for BD to make an estimate of what ID’s decision would be because he doesn’t have ID’s computing power. BD could not estimate ID’s utility function with any more information that BD has access to, so BD’s estimate of ID’s utility function is no better than BD’s own utility function.
First of all, when I (Base Dorikka, BD) am using the phrases “X is moral,” “X is right,” and “I should X” in this comment, I am hypothesizing that a version of myself with a comparatively large amount of computing resources and information (Idealized Dorikka, ID) would come to the conclusion that action X would optimize ID’s utility function in relation to the set of all actions that BD could take in the situation.
Note that I am not stating with that what I say is “moral” would for certain optimize ID’s utility function in relation to the set of all actions that BD could take in the situation—my saying that “X is moral” represents me making a probability estimate that I would do so. It’s key to my understanding here that I don’t actually have any authority to declare something the moral thing, I am just estimating that the probability that X is moral is high.
This is my understanding of the use of such phrases—if you disagree with my conclusions, check if we disagree on how Eliezer uses the words.
Now, BD does not, of course, have access to ID’s computing resources, and so I can only imagine a version of ID a certain level above my own. However, ID would have enough computing resources to imagine an ID+ that was a certain level above him. If ID+ ever came to exist, he would be able to imagine an ID++, and so on.
I think that Eliezer’s metaethics would select option #2, and point out that ID’s reference for what he should do is ID+, not himself.
Edit: I, however, do have the objection that it isn’t meaningful for BD to make an estimate of what ID’s decision would be because he doesn’t have ID’s computing power. BD could not estimate ID’s utility function with any more information that BD has access to, so BD’s estimate of ID’s utility function is no better than BD’s own utility function.