In the above I’ve mostly just raised questions. If you’re interested in what I think is actually the best way to think about issues such as moral weight for AIs, I suggest you read Grounding Value Learning in Evolutionary Psychology: an Alternative Proposal to CEV. But you might want to warm up first with The Terrible, Horrible, No Good, Very Bad Truth About Morality and What To Do About It, and maybe also A Sense of Fairness: Deconfusing Ethics.
In the above I’ve mostly just raised questions. If you’re interested in what I think is actually the best way to think about issues such as moral weight for AIs, I suggest you read Grounding Value Learning in Evolutionary Psychology: an Alternative Proposal to CEV. But you might want to warm up first with The Terrible, Horrible, No Good, Very Bad Truth About Morality and What To Do About It, and maybe also A Sense of Fairness: Deconfusing Ethics.