I will answer this from the perspective of extrapolative alignment proposals like Coherent Extrapolated Volition or MetaEthical AI, in which the value system, utility function, decision theory… that is to govern the future, is extrapolated in some way from present-day humans.
The assumption is that an appropriate transhuman notion e.g. of good and evil, can be extrapolated from the less contingent parts, of human notions of good and evil. In some cases this will consist of identifying deeper principles that have something to say about all possible forms of life and mind, and not just something to say about the situations with which humanity is historically familiar.
For example, principles like: maximize net pleasure, minimize net pain; allow each agent to do whatever it wants, except insofar as it interferes with the freedom of other agents; “from each according to its abilities, to each according to its needs”… Each of these is the product of human ethical reflection and observation. Yet none of those is inherently anthropocentric, and each could be the basis of a moral-political order encompassing a diversity of entities far beyond anything that exists today.
I will answer this from the perspective of extrapolative alignment proposals like Coherent Extrapolated Volition or MetaEthical AI, in which the value system, utility function, decision theory… that is to govern the future, is extrapolated in some way from present-day humans.
The assumption is that an appropriate transhuman notion e.g. of good and evil, can be extrapolated from the less contingent parts, of human notions of good and evil. In some cases this will consist of identifying deeper principles that have something to say about all possible forms of life and mind, and not just something to say about the situations with which humanity is historically familiar.
For example, principles like: maximize net pleasure, minimize net pain; allow each agent to do whatever it wants, except insofar as it interferes with the freedom of other agents; “from each according to its abilities, to each according to its needs”… Each of these is the product of human ethical reflection and observation. Yet none of those is inherently anthropocentric, and each could be the basis of a moral-political order encompassing a diversity of entities far beyond anything that exists today.