I may hunt through my old comments to find the ones where I was working this through a few years back, but I think I would summarize mine as:
A target’s CEV does not uniquely exist: there’s a large set of equally legitimate extrapolation processes that can be performed on a given target and their results don’t converge, because human values and preferences change based on the events we experience. So many of the above options are non-mutually-exclusively true, but that’s not a particularly interesting fact.
Even for a fixed extrapolation-method, CEV(A + B) bears no predictable relationship to CEV(A) and CEV(B), for the same reason: being part of group A is a different environment than being part of group (A+B) and results in different values and preferences. So the CEV of a group containing me varies greatly depending on the group.
That X is part of (one) CEV for A doesn’t guarantee that A would endorse X given the choice; in fact, if A is human it’s unlikely, since (again) human values aren’t fixed. So implementing our extrapolated volition—any of them—will likely mean violating our current preferences.
I may hunt through my old comments to find the ones where I was working this through a few years back, but I think I would summarize mine as:
A target’s CEV does not uniquely exist: there’s a large set of equally legitimate extrapolation processes that can be performed on a given target and their results don’t converge, because human values and preferences change based on the events we experience. So many of the above options are non-mutually-exclusively true, but that’s not a particularly interesting fact.
Even for a fixed extrapolation-method, CEV(A + B) bears no predictable relationship to CEV(A) and CEV(B), for the same reason: being part of group A is a different environment than being part of group (A+B) and results in different values and preferences. So the CEV of a group containing me varies greatly depending on the group.
That X is part of (one) CEV for A doesn’t guarantee that A would endorse X given the choice; in fact, if A is human it’s unlikely, since (again) human values aren’t fixed. So implementing our extrapolated volition—any of them—will likely mean violating our current preferences.