Minimal alignment is indeed a necessary premise, the argument doesn’t work without that, and a lot of the post is gesturing at evidence for it. In principle even uploading could be skipped by a sufficiently stingy superintelligence, and humanity rebooted later (when no longer inconvenient) from mere genetic data, an initial fake society of AGIs pretending to be humans, and Internet archives to restore the cultural state (or in reality, a superintelligent simulation of such a process that won’t be as clunky as this exploratory engineering sketch). This kills the presently living humans, but preserves the future of humanity.
The quoted text is about quality of life that the permanently disempowered future of humanity might expect, and why permanent disempowerment in particular is the sole privileged malus it must suffer. It’s different from all other hypothetical ills in that amending it would take the kinds of resources that are actually relevant even for superintelligence in the long run.
Minimal alignment is indeed a necessary premise, the argument doesn’t work without that, and a lot of the post is gesturing at evidence for it. In principle even uploading could be skipped by a sufficiently stingy superintelligence, and humanity rebooted later (when no longer inconvenient) from mere genetic data, an initial fake society of AGIs pretending to be humans, and Internet archives to restore the cultural state (or in reality, a superintelligent simulation of such a process that won’t be as clunky as this exploratory engineering sketch). This kills the presently living humans, but preserves the future of humanity.
The quoted text is about quality of life that the permanently disempowered future of humanity might expect, and why permanent disempowerment in particular is the sole privileged malus it must suffer. It’s different from all other hypothetical ills in that amending it would take the kinds of resources that are actually relevant even for superintelligence in the long run.