In a nutshell, Eliezer’s metaethics says you should maximize your preferences whatever they may be, or rather, you should_you maximize your preferences, but of course you should_me maximize my preferences. (Note that I said preferences and not utility function.
Eliezer is a bit more aggressive in the use of ‘should’. What you are describing as should Eliezer has declared to be would_want while ‘should’ is implicitly would_want, with no allowance for generic instantiation. That is he is comfortable answering “What should a Paperclip Maximiser do when faced with Newcomb’s problem?” with “Rewrite itself to be an FAI”.
There have been rather extended (and somewhat critical) discussions in comment threads of Eliezer’s slightly idiosyncratic usage of ‘should’ and related terminology but I can’t recall where. I know it was in a thread not directly related to the subject!
You’re right about Eliezer’s semantics. Count me as one of those who thought his terminology was confusing, which is why I don’t use it when I try to describe the theory to anyone else.
Are you sure? I thought “should” could mean would_want. Note I could follow this by saying “That is he is comfortable answering “What should a Paperclip Maximiser do when faced with Newcomb’s problem?” with “Rewrite itself to be an FAI”.”, but that would be affirming the consequent ;-), i.e. I know he says such a thing, but my and your formulation both plausibly explain it, as far as I know.
Eliezer is a bit more aggressive in the use of ‘should’. What you are describing as should Eliezer has declared to be would_want while ‘should’ is implicitly would_want, with no allowance for generic instantiation. That is he is comfortable answering “What should a Paperclip Maximiser do when faced with Newcomb’s problem?” with “Rewrite itself to be an FAI”.
There have been rather extended (and somewhat critical) discussions in comment threads of Eliezer’s slightly idiosyncratic usage of ‘should’ and related terminology but I can’t recall where. I know it was in a thread not directly related to the subject!
You’re right about Eliezer’s semantics. Count me as one of those who thought his terminology was confusing, which is why I don’t use it when I try to describe the theory to anyone else.
Are you sure? I thought “should” could mean would_want. Note I could follow this by saying “That is he is comfortable answering “What should a Paperclip Maximiser do when faced with Newcomb’s problem?” with “Rewrite itself to be an FAI”.”, but that would be affirming the consequent ;-), i.e. I know he says such a thing, but my and your formulation both plausibly explain it, as far as I know.