Strong upvote because of the beauty and approximate-correctness of your writing, but I disagree on certain points. For one thing, I think inefficiency is a sin and people do not have the right to choose physical embodiment when they can have the same exact qualia while uploaded and use far less energy and matter.
For another, I think suffering is absolutely bad and only someone who is not presently suffering (or who has traumas that make them attached to their suffering) could conceivably disagree. There is no acceptable amount of suffering, particularly of the kind that isn’t chosen on purpose for some reason. (Although I am unsure that is acceptable either.) Nonhuman presophont beings are unable to make such choices and therefore every mote of suffering they experience is absolutely wrong and must be abolished.
The true benevolent singularity would embark on David Pearce style paradise engineering, liberating not only all humans, but all living things from death, suffering, and coercion—as well as, I think, from ignorance, including presophonce itself—as I do not think it is moral for a being to have less than maximal possible agency consistent with all other agencies, which means it is immoral to allow a being to continue not possessing sapience and sophonce any longer than absolutely necessary for the safety of all others. This uplift probably would be gradual, but the abolition of suffering would be immediate. The predator’s belly would be filled with artificial meat, the disease-causing microbes would be re-engineered into mutualists, even the smallest twinges of pain eliminated.
Otherwise, I agree with you. There is no such thing as coherent extrapolated volition. Each entity has its own will and has the right to live in their own way, in accordance with their preferences, while being prevented from interfering with any other. But as I said at first: preferences are only legitimate as regards subjective experience. You don’t, imo, have the right to decide your substrate: you really are made of atoms that the Singularity could use more efficiently for something else—such as expanding your consciousness, or simulating more unique, joyful, loving entities.
The power imbalance between modern civilization and the Sentinelese is so profound that one could easily imagine it being a crude imitation of what to expect from a superintelligence and humanity. The Sentinelese offer virtually no benefit to India or the West, and in fact occupy an island that could be used for other purposes. The Sentinelese have actively killed people who have gotten too close to them. They live pseudo-Paleolithic lives of naturalistic peril. They lack modern medicine and live on hunter-gatherer instincts. They are not even capable of creating fire by themselves. By all metrics, the Sentinelese serve no purpose and should either be wiped out or forcibly assimilated into modern culture.
And yet we don’t do this because we respect their autonomy and right to live as they so choose, even if it’s a less “civilized” way of life. If members of the Sentinelese choose to integrate into modern society, we’re more than open for them to join us. But the group is not completely uncontacted; they are aware we exist. They simply choose to stay insulated, even with the knowledge that we have advanced technology.
This is what alignment looks like. It may not be “pretty” or “benevolent” but we as a society are aligned to the values of the Sentinelese.
If we did have greater benevolence, we would never tolerate the Sentinelese living this way, but it would come at the cost of their autonomy and right to live as they wish. Indeed, we did have such a mindset once upon a time, and it is widely seen as “colonialist” and detrimental to the Sentinelese by modern standards. If the Sentinelese choose to live this way, who are we to decide for them, even if we know better?
It’s entirely possible that an aligned superintelligence would have very similar thoughts about humanity as a whole.
That’s certainly better than extinction. But Christlike benevolence is the thing to aim for. Call it colonialism if you want—I think the Sentinelese would be better off living more like the rest of us, too.
Strong upvote because of the beauty and approximate-correctness of your writing, but I disagree on certain points. For one thing, I think inefficiency is a sin and people do not have the right to choose physical embodiment when they can have the same exact qualia while uploaded and use far less energy and matter.
For another, I think suffering is absolutely bad and only someone who is not presently suffering (or who has traumas that make them attached to their suffering) could conceivably disagree. There is no acceptable amount of suffering, particularly of the kind that isn’t chosen on purpose for some reason. (Although I am unsure that is acceptable either.) Nonhuman presophont beings are unable to make such choices and therefore every mote of suffering they experience is absolutely wrong and must be abolished.
The true benevolent singularity would embark on David Pearce style paradise engineering, liberating not only all humans, but all living things from death, suffering, and coercion—as well as, I think, from ignorance, including presophonce itself—as I do not think it is moral for a being to have less than maximal possible agency consistent with all other agencies, which means it is immoral to allow a being to continue not possessing sapience and sophonce any longer than absolutely necessary for the safety of all others. This uplift probably would be gradual, but the abolition of suffering would be immediate. The predator’s belly would be filled with artificial meat, the disease-causing microbes would be re-engineered into mutualists, even the smallest twinges of pain eliminated.
Otherwise, I agree with you. There is no such thing as coherent extrapolated volition. Each entity has its own will and has the right to live in their own way, in accordance with their preferences, while being prevented from interfering with any other. But as I said at first: preferences are only legitimate as regards subjective experience. You don’t, imo, have the right to decide your substrate: you really are made of atoms that the Singularity could use more efficiently for something else—such as expanding your consciousness, or simulating more unique, joyful, loving entities.
There is a difference between a truly benevolent superintelligence and an aligned superintelligence.
Alignment doesn’t necessarily mean Christlike benevolence.
Indeed, as I posited up above, we actually have a real-life analog for what “alignment” looks like: the Sentinelese
https://en.wikipedia.org/wiki/Sentinelese
The power imbalance between modern civilization and the Sentinelese is so profound that one could easily imagine it being a crude imitation of what to expect from a superintelligence and humanity. The Sentinelese offer virtually no benefit to India or the West, and in fact occupy an island that could be used for other purposes. The Sentinelese have actively killed people who have gotten too close to them. They live pseudo-Paleolithic lives of naturalistic peril. They lack modern medicine and live on hunter-gatherer instincts. They are not even capable of creating fire by themselves. By all metrics, the Sentinelese serve no purpose and should either be wiped out or forcibly assimilated into modern culture.
And yet we don’t do this because we respect their autonomy and right to live as they so choose, even if it’s a less “civilized” way of life. If members of the Sentinelese choose to integrate into modern society, we’re more than open for them to join us. But the group is not completely uncontacted; they are aware we exist. They simply choose to stay insulated, even with the knowledge that we have advanced technology.
This is what alignment looks like. It may not be “pretty” or “benevolent” but we as a society are aligned to the values of the Sentinelese.
If we did have greater benevolence, we would never tolerate the Sentinelese living this way, but it would come at the cost of their autonomy and right to live as they wish. Indeed, we did have such a mindset once upon a time, and it is widely seen as “colonialist” and detrimental to the Sentinelese by modern standards. If the Sentinelese choose to live this way, who are we to decide for them, even if we know better?
It’s entirely possible that an aligned superintelligence would have very similar thoughts about humanity as a whole.
That’s certainly better than extinction. But Christlike benevolence is the thing to aim for. Call it colonialism if you want—I think the Sentinelese would be better off living more like the rest of us, too.