The question is: extrapolated volition of whom? In the case of thinking about whether to create super happy AIs that replace us (A) or not (B), this would presumably be our current human extrapolated volition. So it wouldn’t take interests of non-existing AIs into account. And in the case of asking whether colonization of America was good or bad, we would have to consider the extrapolated volition of the humans alive at the time.
It’s a bit tricky. I don’t super feel like I owe the competitors to my distant ancestors in the primordial soup consideration in humanity’s CEV, though I am also not enormously confident that I definitely don’t.
Definitely agree that in this case you consider the value of the people who you took the opportunity to reproduce from (though also ultimately I will also at least somewhat bite the bullet that my values might diverge from theirs and in as much as we are in a fully zero-sum competition I would like my values to win out, though overall principles of fairness and justice definitely compel me to give them a non-trivial chunk of the Lightcone).
The question is: extrapolated volition of whom? In the case of thinking about whether to create super happy AIs that replace us (A) or not (B), this would presumably be our current human extrapolated volition. So it wouldn’t take interests of non-existing AIs into account. And in the case of asking whether colonization of America was good or bad, we would have to consider the extrapolated volition of the humans alive at the time.
It’s a bit tricky. I don’t super feel like I owe the competitors to my distant ancestors in the primordial soup consideration in humanity’s CEV, though I am also not enormously confident that I definitely don’t.
Definitely agree that in this case you consider the value of the people who you took the opportunity to reproduce from (though also ultimately I will also at least somewhat bite the bullet that my values might diverge from theirs and in as much as we are in a fully zero-sum competition I would like my values to win out, though overall principles of fairness and justice definitely compel me to give them a non-trivial chunk of the Lightcone).