The arguments around CEV suggest that these moral theories ought to converge.
In the practical sense, only something in particular can be done with the world, so if “morality” is taken to refer to the goal given to a world-optimizing AI, it should be something specific by construction. If we take “morality” as given by the data of individual people, we can define personal moralities for each of them that would almost certainly be somewhat different from each other. Given the task of arriving at a single goal for the world, it might prove useful to exploit the similarities between personal moralities, or to sidestep this concept altogether, but eventual “convergence” is more of a design criterion than a prediction. In a world that had both humans and pebblesorters in it, arriving at a single goal would still be an important problem, even though we wouldn’t expect these goals to “naturally” converge under reflection.
In the practical sense, only something in particular can be done with the world, so if “morality” is taken to refer to the goal given to a world-optimizing AI, it should be something specific by construction. If we take “morality” as given by the data of individual people, we can define personal moralities for each of them that would almost certainly be somewhat different from each other. Given the task of arriving at a single goal for the world, it might prove useful to exploit the similarities between personal moralities, or to sidestep this concept altogether, but eventual “convergence” is more of a design criterion than a prediction. In a world that had both humans and pebblesorters in it, arriving at a single goal would still be an important problem, even though we wouldn’t expect these goals to “naturally” converge under reflection.