Phil Goetz, why should I care what sort of creatures the universe “tends to produce”? What makes this a moral argument that should move me? Do you think that most creatures the universe produces must inevitably evolve to be moved by such an argument?
I stated the reason:
We MUST make this meta-level argument that the universe inherently produces creatures with pretty-valuable values. We have no other way of claiming to be better than pebble-sorters.
I don’t think that we can argue for our framework of ideas from within our framework of ideas. If we continue to insist that we are better than pebble-sorters, we can justify it only by claiming that the processes that lead to our existence tend to produce good outcomes, whereas the hypothetical pebble-sorters are chosen from a much larger set of possible beings, with a much lower average moral acceptability.
A problem with this is that all sorts of insects and animals exist with horrifying “moral systems”. We might convince ourselves that morals improve as a society becomes more complex. (That’s just a thought in postscript.)
One possible conclusion—not one that I have reached, but one that you might conclude if the evidence comes out a certain way—is that the right thing to do is not to make any attempt to control the morals of AIs, because general evolutionary processes may be better at designing morals than we are.
Thinking about this post leads me to conclude that CEV is not the most right thing to do. There may be a problem with my reasoning, in that it could also be used by pebble-sorters to justify continued pebble-sorting. However, my reasoning includes the consequence that pebble-sorters are impossible, so that is a non-issue.
Think about our assumption that we are in fact better than pebble-sorters. It seems impossible for us to construct an argument concluding this, because any argument we make presumes the values we are trying to conclude.
Yet we continue to use the pebble-sorters, not as an example of another, equally-valid ethical system, but as an example of something wrong.
We can justify this by making a meta-level argument that the universe is biased to produce organisms with relatively valuable values. (I’m worried about the semantics of that statement, but let me continue.) Pebble-sorting, and other futile endeavors, are non-adaptive, and will lose any evolutionary race to systems that generate increased complexity (from some energy input).
We MUST make this meta-level argument that the universe inherently produces creatures with pretty-valuable values. We have no other way of claiming to be better than pebble-sorters.
Given this, we could use CEV to construct AIs… but we can also try to understand WHY the universe produces good values. Once we understand that, we can use the universe’s rules to direct the construction of AIs. This could result in AIs with wildly different values than our own, but it may be more likely to result in non-futile AIs, or to produce more-optimal AIs (in terms of their values).
It may, in fact, be difficult or impossible to construct AIs that aren’t eventually subject to the universe’s benevolent, value-producing bias—since these AIs will be in the universe. But we have seen in human history that, although there are general forces causing societies with some of our values to prosper, we nonetheless find societies in local minima in which they are in continual warfare, pain, and poverty. So some effort on our part may increase the odds of, or the decrease the time until, a good result.