I think an important distinction is one you don’t make, where humans remain alive, but lose cosmic endowment. AGIs are probably not starting out as mature optimizers, so their terminal values are going to follow from a process of volition extrapolation, similar to that needed for humans. If they happen to hold enough human values from training on texts and other media, and don’t get tuned into something completely alien, it’s likely they at least give us some computational welfare sufficient for survival.
A less likely possibility is that somehow processes of volition extrapolation converge across different humans/AGIs to some currently illegible genericterminalvalues, in which case AGIs’ use of cosmic endowment is going to be valuable to humanity’s CEV as well, and human people are more likely to meaningfully participate.
I think an important distinction is one you don’t make, where humans remain alive, but lose cosmic endowment. AGIs are probably not starting out as mature optimizers, so their terminal values are going to follow from a process of volition extrapolation, similar to that needed for humans. If they happen to hold enough human values from training on texts and other media, and don’t get tuned into something completely alien, it’s likely they at least give us some computational welfare sufficient for survival.
A less likely possibility is that somehow processes of volition extrapolation converge across different humans/AGIs to some currently illegible generic terminal values, in which case AGIs’ use of cosmic endowment is going to be valuable to humanity’s CEV as well, and human people are more likely to meaningfully participate.