Prase, Chris, I don’t understand. Eliezer’s example is set up in such a way that, regardless of what the paperclip maximizer does, defecting gains one billion lives and loses two paperclips.
Basically, we’re being asked to choose between a billion lives and two paperclips (paperclips in another universe, no less, so we can’t even put them to good use).
The only argument for cooperating would be if we had reason to believe that the paperclip maximizer will somehow do whatever we do. But I can’t imagine how that could be true. Being a paperclip maximizer, it’s bound to defect, unless it had reason to believe that we would somehow do whatever it does. I can’t imagine how that could be true either.
Or am I missing something?
This post hits me far more strongly than the previous ones on this subject.
I think your main point is that it’s positively dangerous to believe in an objective account of morality, if you’re trying to build an AI. Because you will then falsely believe that a sufficiently intelligent AI will be able to determine the correct morality—so you don’t have to worry about programming it to be friendly (or Friendly).
I’m sure you’ve mentioned this before, but this is more forceful, at least to me. Thanks.
Personally, even though I’ve mentioned that I thought there might be an objective basis for morality, I’ve never believed that every mind (or even a large fraction of minds) would be able to find it. So I’m in total agreement that we shouldn’t just assume a superintelligent AI would do good things.
In other words, this post drives home to me that, pragmatically, the view of morality you propose is the best one to have, from the point of view of building an AI.