Assumption 1: The paperclip maximizer does not have magical information about my intentions, so changing my intentions will not change its decision.
I prefer (D,C) to (C,C), and I prefer (D,D) to (C,D). If I were to know the paperclip maximizer’s choice ahead of time, it would not change my choice; and if the paperclip maximizer knew my choice ahead of time, it would not change its choice.
Here’s the real question: Are you willing to sacrifice two paperclips to save a billion human lives? Is the paperclip maximizer willing to sacrifice two billion human lives to create one paperclip?
Would it be rational to execute the reverse, even if possible: For the humans to sacrifice human lives to create paperclips, on the possibility that the paperclip maximizer would sacrifice paperclips to save humans? Given that they are better at saving humans than we are, and that we are better at making paperclips than they are, it would appear to be the case.
Assumption 1: The paperclip maximizer does not have magical information about my intentions, so changing my intentions will not change its decision.
I prefer (D,C) to (C,C), and I prefer (D,D) to (C,D). If I were to know the paperclip maximizer’s choice ahead of time, it would not change my choice; and if the paperclip maximizer knew my choice ahead of time, it would not change its choice.
Here’s the real question: Are you willing to sacrifice two paperclips to save a billion human lives? Is the paperclip maximizer willing to sacrifice two billion human lives to create one paperclip?
Would it be rational to execute the reverse, even if possible: For the humans to sacrifice human lives to create paperclips, on the possibility that the paperclip maximizer would sacrifice paperclips to save humans? Given that they are better at saving humans than we are, and that we are better at making paperclips than they are, it would appear to be the case.