So I want to poke at you and see if I’m understanding you correctly.
First, are you just talking about strong moral essentialism (morals are powered by real, possibly observable, facts or processes, the are causally connected to moral judgements) here or all moral realism (moral facts exist, even if they are unknowable)?
Second, what makes you think a moral realist would not be in favor of AI value learning such that you need to argue for it?
I think there are ways in which believing in moral realism may make you sloppy about value learning and is more likely to result in learning designs that will result in learning values that we would, in retrospect, not endorse, but I don’t see that as suggesting a moral realist would be against value learning (preprint on this point). In fact, I expect them to be for it, only that they will expect values to naturally converge no matter the input where an anti-realist would except input to matter a lot. The standard objection from an antirealist would be “seems like data matters a lot so far to outcomes” and the standard realist reply might be “need more data or less biased data”.
Yeah, I’m not 100% my caricature of a person actually exists or is worth addressing. They’re mostly modeled on Robert Nozick, who is dead and cannot be reached for comment on value learning. But I had most of these thoughts and the post was really easy to write, so I decided to post it. Oh well :)
The person I am hypothetically thinking about is not very systematic—on average, they would admit that they don’t know where morality comes from. But they feel like they learn about morality by interacting in some mysterious way with an external moral reality, and that an AI is going to be missing something important—maybe even be unable to do good—if they don’t do that too. (So 90% overlap with your description of strong moral essentialism.)
I think these people plausibly should be for value learning, but are going to be dissatisfied with it and feel like it sends the wrong philosophical message.
So I want to poke at you and see if I’m understanding you correctly.
First, are you just talking about strong moral essentialism (morals are powered by real, possibly observable, facts or processes, the are causally connected to moral judgements) here or all moral realism (moral facts exist, even if they are unknowable)?
Second, what makes you think a moral realist would not be in favor of AI value learning such that you need to argue for it?
I think there are ways in which believing in moral realism may make you sloppy about value learning and is more likely to result in learning designs that will result in learning values that we would, in retrospect, not endorse, but I don’t see that as suggesting a moral realist would be against value learning (preprint on this point). In fact, I expect them to be for it, only that they will expect values to naturally converge no matter the input where an anti-realist would except input to matter a lot. The standard objection from an antirealist would be “seems like data matters a lot so far to outcomes” and the standard realist reply might be “need more data or less biased data”.
Yeah, I’m not 100% my caricature of a person actually exists or is worth addressing. They’re mostly modeled on Robert Nozick, who is dead and cannot be reached for comment on value learning. But I had most of these thoughts and the post was really easy to write, so I decided to post it. Oh well :)
The person I am hypothetically thinking about is not very systematic—on average, they would admit that they don’t know where morality comes from. But they feel like they learn about morality by interacting in some mysterious way with an external moral reality, and that an AI is going to be missing something important—maybe even be unable to do good—if they don’t do that too. (So 90% overlap with your description of strong moral essentialism.)
I think these people plausibly should be for value learning, but are going to be dissatisfied with it and feel like it sends the wrong philosophical message.
What does it mean for a reality to be moral?