I agree with most of this post, and in fact my recent posts (at my blog, not here) imply something similar. But I think there is a mistaken idea in this particular statement: “We want a safe way to resolve the under-determination in human values, a task that gets more and more difficult as we move away from the usual world of today and into the hypothetical world that a superpowered AI could build.”
It looks like you are saying that we need a way to make sure that the future, however distant, will always be somewhat acceptable to current humans. But this is impossible in principle, given the fact that things are tending towards the heat death of the universe. What we actually should want is that the universe should move at any particular time towards things that the beings in existence value at that time. Obviously creatures in the future will have different values, and given a long enough time period, a future will therefore come into existence that we as we are would have no particular interest in. But we also should no particular interest in preventing it from coming into being; that interest comes from a positively unreasonable extrapolation of your current interests.
I agree with most of this post, and in fact my recent posts (at my blog, not here) imply something similar. But I think there is a mistaken idea in this particular statement: “We want a safe way to resolve the under-determination in human values, a task that gets more and more difficult as we move away from the usual world of today and into the hypothetical world that a superpowered AI could build.”
It looks like you are saying that we need a way to make sure that the future, however distant, will always be somewhat acceptable to current humans. But this is impossible in principle, given the fact that things are tending towards the heat death of the universe. What we actually should want is that the universe should move at any particular time towards things that the beings in existence value at that time. Obviously creatures in the future will have different values, and given a long enough time period, a future will therefore come into existence that we as we are would have no particular interest in. But we also should no particular interest in preventing it from coming into being; that interest comes from a positively unreasonable extrapolation of your current interests.