Thanks for this response. One comment about one of your main points: I agree that the tradeoff of number of humans vs. length of life is ambiguous. But to the extent our utility function favors numbers of people over total life span, that makes the second scenario more plausible, whereas if total life span is more important, the first is more plausible.
I agree with you that both the scenarios would be totally unacceptable to me personally, because of my limited altruism. I would badly want to stop it from happening, and I would oppose creating any AI that did it. But I disagree in that I can’t say that any such AI is unfriendly or “evil”. Maybe if I was less egoistic, and had a better capacity to understand the consequences, I really would feel the sacrifice was worth it.
If you would oppose an AI attempting to enforce a CEV that would be detrimental to you, but still classify it as FAI and not evil, then wouldn’t that make you evil?
Obviously this is a matter of definitions, but it still seems to be the logical conclusion.
Thanks for this response. One comment about one of your main points: I agree that the tradeoff of number of humans vs. length of life is ambiguous. But to the extent our utility function favors numbers of people over total life span, that makes the second scenario more plausible, whereas if total life span is more important, the first is more plausible.
I agree with you that both the scenarios would be totally unacceptable to me personally, because of my limited altruism. I would badly want to stop it from happening, and I would oppose creating any AI that did it. But I disagree in that I can’t say that any such AI is unfriendly or “evil”. Maybe if I was less egoistic, and had a better capacity to understand the consequences, I really would feel the sacrifice was worth it.
If you would oppose an AI attempting to enforce a CEV that would be detrimental to you, but still classify it as FAI and not evil, then wouldn’t that make you evil?
Obviously this is a matter of definitions, but it still seems to be the logical conclusion.