I don’t believe in golden rules written in fire etched into the fabric of reality, or divine commands about what everyone should do, but I know there are some weird things that have conscious experiences and memory, and this seems something valuable in itself. Moreover, I don’t see other sources of value at the moment. I guess I’ll do something about it.”
It’s possible for a human to reject the idea that anything is valuable in itself, so is possible for an AI. You are assuming, not arguing for, the idea that the AI must be a moral realist (and that it is going to agree with human realists about whats really valuable , without having the same parochialism).
ETA.
If naturalism is only the claim that the correct metaethics can be discovered by science, then there is no necessary implication that the correct metaethics is natural realism,ie.some things having inherent value. In the contrary, the claims that value is subjective,and that ethical systems evolved or constructed, are naturalistic but not realistic, so anti-realistic naturalism is possible.
It’s possible for a human to reject the idea that anything is valuable in itself, so is possible for an AI. You are assuming, not arguing for, the idea that the AI must be a moral realist (and that it is going to agree with human realists about whats really valuable , without having the same parochialism).
ETA. If naturalism is only the claim that the correct metaethics can be discovered by science, then there is no necessary implication that the correct metaethics is natural realism,ie.some things having inherent value. In the contrary, the claims that value is subjective,and that ethical systems evolved or constructed, are naturalistic but not realistic, so anti-realistic naturalism is possible.