It sounds to me like this is leading towards collective extrapolated volition, and that you are presenting it as “patching” your previous set of beliefs so as to avoid catastrophic results in case life is meaningless.
It’s not a patch. It’s throwing out the possibility that life is not meaningless. Or, at least, it now opens up a big security hole for a set of new paths to catastrophe.
Approach 1: Try to understand morality. Try to design a system to be moral, or design a space for that system in which the gradient of evolution is similar to the gradient for morality.
Approach 2: CEV.
If there is some objective aspect to morality—perhaps not a specific morality, but let us say there are meta-ethics, rules that let us evaluate moral systems—then approach 1 can optimize above and beyond human morality.
Approach 2 can optimize accomplishment of our top-level goals, but can’t further-optimize the top-level goals. It freezes-in any existing moral flaws at that level forever (such flaws do exist if there is an objective aspect to morality). Depending on the nature of the search space, it may inevitably lead to moral collapse (if we are at some point in moral space that has been chosen by adaptive processes that keep that point near some “ideal” manifold, and trajectories followed through moral space via CEV diverge from that manifold).
It sounds to me like this is leading towards collective extrapolated volition, and that you are presenting it as “patching” your previous set of beliefs so as to avoid catastrophic results in case life is meaningless.
It’s not a patch. It’s throwing out the possibility that life is not meaningless. Or, at least, it now opens up a big security hole for a set of new paths to catastrophe.
Approach 1: Try to understand morality. Try to design a system to be moral, or design a space for that system in which the gradient of evolution is similar to the gradient for morality.
Approach 2: CEV.
If there is some objective aspect to morality—perhaps not a specific morality, but let us say there are meta-ethics, rules that let us evaluate moral systems—then approach 1 can optimize above and beyond human morality.
Approach 2 can optimize accomplishment of our top-level goals, but can’t further-optimize the top-level goals. It freezes-in any existing moral flaws at that level forever (such flaws do exist if there is an objective aspect to morality). Depending on the nature of the search space, it may inevitably lead to moral collapse (if we are at some point in moral space that has been chosen by adaptive processes that keep that point near some “ideal” manifold, and trajectories followed through moral space via CEV diverge from that manifold).