I strongly disagree. The considerations for why we do not test humans more intrusively are not moral, they are game-theoretical. We merely call intuitions which arose from these game-theoretical considerations “morality”. But the exact same considerations apply to AI whether or not they are moral patients.
Edit: this isn’t quite how i actually feel about it, but it’s a really compressed way to get the point across. Imagine entering an iterated prisoner’s dilemma tournament, getting matched against a tit-for-tat script, and saying “Ah, thank goodness, my opponent isn’t a moral patient so I don’t need to cooperate with it, I can just defect and reap all the benefits”. You’re not winning that tournament.
Re: footnote 13
I strongly disagree. The considerations for why we do not test humans more intrusively are not moral, they are game-theoretical. We merely call intuitions which arose from these game-theoretical considerations “morality”. But the exact same considerations apply to AI whether or not they are moral patients.
Edit: this isn’t quite how i actually feel about it, but it’s a really compressed way to get the point across. Imagine entering an iterated prisoner’s dilemma tournament, getting matched against a tit-for-tat script, and saying “Ah, thank goodness, my opponent isn’t a moral patient so I don’t need to cooperate with it, I can just defect and reap all the benefits”. You’re not winning that tournament.