It’s even worse than that: 1) “we” know that “our” values now are, at least slightly, different to what they were 10,000 years ago. 2) We have no reason to believe that we are currently at a state of peak, absolute values (whatever that might mean) and therefore expect that, absent SGI, our values will be different in 10,000 years. 3) If we turn over power to an SGI, perfectly aligned with our current values then they will be frozen for the rest of time. Alternatively, if we want it to allow our values to change “naturally” over time it will be compelled to do nothing as doing anything at all would effectively be shaping our values in some direction that we have not specified. 4) Therefore our current values cannot be a sound basis for the utility funtion for an SGI that is not somehow limited in time or scope.
It’s even worse than that:
1) “we” know that “our” values now are, at least slightly, different to what they were 10,000 years ago.
2) We have no reason to believe that we are currently at a state of peak, absolute values (whatever that might mean) and therefore expect that, absent SGI, our values will be different in 10,000 years.
3) If we turn over power to an SGI, perfectly aligned with our current values then they will be frozen for the rest of time. Alternatively, if we want it to allow our values to change “naturally” over time it will be compelled to do nothing as doing anything at all would effectively be shaping our values in some direction that we have not specified.
4) Therefore our current values cannot be a sound basis for the utility funtion for an SGI that is not somehow limited in time or scope.