I interpret the quote to mean that there’s no guarantee that the reflection process converges. Its attractor could be a large, possibly infinite, set of states rather than a single point.
I think that’s possible, but I’m saying we can just pick one of the endpoints (or pick an arbitrary, potentially infinitely-long path towards an endpoint), and most people (the original people, and the people who result from that picking) will probably be fine with that, even if involves making some tough and / or arbitrary choices along the way.
Or, if humans on reflection turn out to never want to make all of those choices, that’s maybe also OK. But we probably need at least one person (or AI) to fully “grow up” into a coherent being, in order to actually do really big stuff, like putting up some guardrails in the universe.
That growing up process (which is hopefully causally descended from deliberate human action at some point far back enough) might involve making some arbitrary and tough choices in order to force it to converge in a reasonable length of time. But those choices seem worth making, because the guardrails are important, and an entity powerful enough to set them up is probably going to run into moral edge cases unavoidably. Better its behavior in those cases be decided by some deliberate process in humans, rather than left to some process even more arbitrary and morally unsatisfying.
CEV also has another problem that gets in the way of practically implementing it: it isn’t embedded. At least in its current form, CEV doesn’t have a way of accounting for side-effects (either physical or decision-theoretic) of the reflection process. When you have to deal with embeddedness, the distinction between reflection and action breaks down and you don’t end up getting endpoints at all. At best, you can get a heuristic approximation.
I interpret the quote to mean that there’s no guarantee that the reflection process converges. Its attractor could be a large, possibly infinite, set of states rather than a single point.
I think that’s possible, but I’m saying we can just pick one of the endpoints (or pick an arbitrary, potentially infinitely-long path towards an endpoint), and most people (the original people, and the people who result from that picking) will probably be fine with that, even if involves making some tough and / or arbitrary choices along the way.
Or, if humans on reflection turn out to never want to make all of those choices, that’s maybe also OK. But we probably need at least one person (or AI) to fully “grow up” into a coherent being, in order to actually do really big stuff, like putting up some guardrails in the universe.
That growing up process (which is hopefully causally descended from deliberate human action at some point far back enough) might involve making some arbitrary and tough choices in order to force it to converge in a reasonable length of time. But those choices seem worth making, because the guardrails are important, and an entity powerful enough to set them up is probably going to run into moral edge cases unavoidably. Better its behavior in those cases be decided by some deliberate process in humans, rather than left to some process even more arbitrary and morally unsatisfying.
CEV also has another problem that gets in the way of practically implementing it: it isn’t embedded. At least in its current form, CEV doesn’t have a way of accounting for side-effects (either physical or decision-theoretic) of the reflection process. When you have to deal with embeddedness, the distinction between reflection and action breaks down and you don’t end up getting endpoints at all. At best, you can get a heuristic approximation.