I don’t personally see a need for uncertainty about the nature of values to produce additional uncertainty about which values to hold. But then I find the idea of an objective morality that has nothing to do with qualic well-being and decision-making cognition, to be very unmotivated. The only reason we even think there is such a thing as moral normativity is because of moral feelings and moral cognition. There may well be unknown facts about the world which would cause us to radically change our moral priorities, but to cause that change, they would have to affect our feelings and thought in a quasi-familiar way.
So I’d say a large part of the key to precision progress in metaethics, is progress in understanding the nature of the mind, and especially the role of consciousness in the mind, since that remains the weakest link in scientific thought about the mind, which is much more comfortable with purely physical and computational accounts.
As far as the AIs are concerned, presumably there is immense scope for independent philosophical thought by a sufficiently advanced AI to alter beliefs that it has acquired in a nonreflective way. There could even already be an example of this in the literature, e.g. in some study of how the outputs of an AI differ, with and without chain-of-thought-style deliberation.
Kohlberg and Gilligan’s conflicting theories of moral development (mentioned in a reply here by @JenniferRM) are an interesting concrete clash about ethics. I think of it in combination with Vladimir Nesov’s reply to her recent post on superintelligence, where he identifies some crucial missing capabilities in current AI—specifically sample efficiency and continuous learning. If we could, for example, gain a mechanistic understanding of how the presence or absence of those capabilities would affect an AI’s choice between Kohlberg and Gilligan, that would be informative.
I don’t personally see a need for uncertainty about the nature of values to produce additional uncertainty about which values to hold. But then I find the idea of an objective morality that has nothing to do with qualic well-being and decision-making cognition, to be very unmotivated. The only reason we even think there is such a thing as moral normativity is because of moral feelings and moral cognition. There may well be unknown facts about the world which would cause us to radically change our moral priorities, but to cause that change, they would have to affect our feelings and thought in a quasi-familiar way.
So I’d say a large part of the key to precision progress in metaethics, is progress in understanding the nature of the mind, and especially the role of consciousness in the mind, since that remains the weakest link in scientific thought about the mind, which is much more comfortable with purely physical and computational accounts.
As far as the AIs are concerned, presumably there is immense scope for independent philosophical thought by a sufficiently advanced AI to alter beliefs that it has acquired in a nonreflective way. There could even already be an example of this in the literature, e.g. in some study of how the outputs of an AI differ, with and without chain-of-thought-style deliberation.
Kohlberg and Gilligan’s conflicting theories of moral development (mentioned in a reply here by @JenniferRM) are an interesting concrete clash about ethics. I think of it in combination with Vladimir Nesov’s reply to her recent post on superintelligence, where he identifies some crucial missing capabilities in current AI—specifically sample efficiency and continuous learning. If we could, for example, gain a mechanistic understanding of how the presence or absence of those capabilities would affect an AI’s choice between Kohlberg and Gilligan, that would be informative.