I feel like, the weirder things get, the more difficult it will be even for humans to make judgments about what constitutes “death, body harm, or civilization destruction.”
Death: is mind-uploading into a computer and/or a brain-in-a-vat death, or transcendence? What about a person who becomes like a prostheticphile character in Rimworld, whose body (and maybe even brain) are more prosthetic enhancement than original human (kind of like Darth Vader, or the Ship of Theseus). At what point do we say that the original person has “died”? For that matter, what counts as “alive”? Fetuses?
Body harm: even today, people disagree over whether transgender transitioning surgeries count as body harm or body enhancement. Ditto with some of the more ambitious types of plastic surgery, or “height enhancement” that involves ambitious procedures like lengthening leg bones. Ditto for synthetic hormones. Is an ASI supposed to listen to progressives or conservatives on these issues?
Civilization destruction: are we already destroying our civilization? We demolished large parts of our streetcar-oriented civilization (including entire neighborhoods, rail lines, etc.) to make way for automobile-centric civilization. Was that a good thing? Was that a net-increase in civilization? Is “wokeism” a net-increase in civilization or a destruction of “Western Civilization”? Which threatens our industrial civilization more: carbon emissions, or regulating carbon emissions?
If we define civilization as just, “we live in cities and states and have division of labor,” then we might be arbitrarily closing off certain appealing possibilities. For example, imagine a future where humans get to live in something resembling their ancestral environment (beautiful, pristine nature), which gives us all of the reward signals of that environment that we are primed to relish, except we also have self-replicating nanobots to make sure that food is always in plentiful supply for hunting/gathering, diseases/insects/animals that are dangerous to humans are either eradicated or kept carefully in check, nanobots repair human cellular machinery so that we live to be 800+ years old on average, etc. That’s a kind of “destruction of civilization” that I might even embrace! (I’d have to think about for a while because it’s still pretty weird, but I wouldn’t rule it out automatically).
I feel like, the weirder things get, the more difficult it will be even for humans to make judgments about what constitutes “death, body harm, or civilization destruction.”
Death: is mind-uploading into a computer and/or a brain-in-a-vat death, or transcendence? What about a person who becomes like a prostheticphile character in Rimworld, whose body (and maybe even brain) are more prosthetic enhancement than original human (kind of like Darth Vader, or the Ship of Theseus). At what point do we say that the original person has “died”? For that matter, what counts as “alive”? Fetuses?
Body harm: even today, people disagree over whether transgender transitioning surgeries count as body harm or body enhancement. Ditto with some of the more ambitious types of plastic surgery, or “height enhancement” that involves ambitious procedures like lengthening leg bones. Ditto for synthetic hormones. Is an ASI supposed to listen to progressives or conservatives on these issues?
Civilization destruction: are we already destroying our civilization? We demolished large parts of our streetcar-oriented civilization (including entire neighborhoods, rail lines, etc.) to make way for automobile-centric civilization. Was that a good thing? Was that a net-increase in civilization? Is “wokeism” a net-increase in civilization or a destruction of “Western Civilization”? Which threatens our industrial civilization more: carbon emissions, or regulating carbon emissions?
If we define civilization as just, “we live in cities and states and have division of labor,” then we might be arbitrarily closing off certain appealing possibilities. For example, imagine a future where humans get to live in something resembling their ancestral environment (beautiful, pristine nature), which gives us all of the reward signals of that environment that we are primed to relish, except we also have self-replicating nanobots to make sure that food is always in plentiful supply for hunting/gathering, diseases/insects/animals that are dangerous to humans are either eradicated or kept carefully in check, nanobots repair human cellular machinery so that we live to be 800+ years old on average, etc. That’s a kind of “destruction of civilization” that I might even embrace! (I’d have to think about for a while because it’s still pretty weird, but I wouldn’t rule it out automatically).