Clearly the paperclip maximizer should just let us have all of substance S; but a paperclip maximizer doesn’t do what it should, it just maximizes paperclips.
I sometimes feel that nitpicking is the only contribution I’m competent to make around here, so… here you endorsed Steven’s formulation of what “should” means; a formulation which doesn’t allow you to apply the word to paperclip maximizers.
I agree: Defect!
Clearly the paperclip maximizer should just let us have all of substance S; but a paperclip maximizer doesn’t do what it should, it just maximizes paperclips.
I sometimes feel that nitpicking is the only contribution I’m competent to make around here, so… here you endorsed Steven’s formulation of what “should” means; a formulation which doesn’t allow you to apply the word to paperclip maximizers.