Eliezer_Yudkowsky wrote: “We don’t want the AI’s models of people to be people—we don’t want conscious minds trapped helplessly inside it. So we need how to tell that something is definitely not a person, and in this case, maybe we would like the AI itself to not be a person, which would simplify a lot of ethical issues if we could pull it off. Creating a new intelligent species is not lightly to be undertaken from a purely ethical perspective; if you create a new kind of person, you have to make sure it leads a life worth living.”
I do want AI models of people to be people. If an AI model of a human lacks the important part of the person that we call emotions, there is the risk that it will be a dangerously faulty model of that person. If it can’t suffer, it can’t understand why humans so eagerly want to get rid of suffering, and so it may in many situations think it knows better than us what is most important to achieve. It may regard the diminishing of suffering in humans just as irrational as we would regard turning the planet into paper clips.
From a utilitarian perspective, even from a negative utilitarian perspective, building AI models of humans and equipping then with human emotions and ability to suffer would not necessarily be unethical. It might be what would in the long run minimize the suffering. Trying to minimize suffering too fast can be dangerous; it might take away something we really value, along with the suffering, even if we don’t value the suffering in and of itself. Future superhuman AI will know how to get rid of suffering in a completely safe way, once they have become in every relevant way immensely smarter than humans—but perhaps only if the first transhuman AI we build are equipped with the ability to suffer just like humans. You often have to know a problem well before you can safely solve it, and you can’t really know suffering unless you actually feel it. So let’s not hesitate to create AI that can suffer (once we can create such AI).
Happiness will grow immensely and suffering will and should be abolished later on, but let’s not rush it during the process of creating AI models of humans.
mtraven, Why we are “bothering to be rational or to do anything at all” (rather than being nihilists), if nihilism seems likely to be valid? Well, as long as there is a chance, say, only a .0000000000000001 chance, that nihilism is invalid, there is nothing to lose and possibly something to gain from assuming that nihilism is invalid. This refutes nihilism completely as a serious alternative.
I think basically the same is true about Yudkowsky’s fear that there are infinitely many copies of each person. Even if there is only a .0000000000000001 chance that there are only finitely many copies of each of us, we should assume that that is the case, since that is the only type of scenario where there can be anything to gain or lose, and thus the only possible type of scenario that might be a good idea to assume to be the case.
That is, given the assumption that one cannot affect infinite amounts by adding, no matter how much one adds. To this, I am an agnostic, if not an atheist. For example, adding an infinite amount A to an infinite amount A can, I think, make 2A rather than 1A. Ask yourself which you would prefer: 1) Being happy one day per year and suffer the rest of the time of each year, for an infinite number of years, or 2) The other way around? Would you really not care which of these two would happen?
You would. Note that this is the case even when you realize that a year is only finitely more than a day, meaning that each of alternatives 1 and 2 would give you infinitely much happiness and infinitely much suffering. This strongly suggests that adding an infinite amount A to an infinite amount A produces more than A. Then why wouldn’t also adding a finite amount B to an infinite amount A produce more than A? I would actually suggest that, even given classical utilitarianism, my life would not be worthless just because there are infinitely much happiness and infinitely much suffering in the world with or without me. Each person’s finite amount of happiness must be of some value regardless of the existence of infinite amounts of happiness elsewhere. I find this to be plausible, because were it not for precisely the individual finite beings with their finite amounts of happiness each, there would be no infinite sums of happiness in the universe. If every single one of all of the universe’s infinitely many, each finitely happy, being’s happiness were worthless, the infinite sum of their happiness would have to be worthless too. And that an infinite sum of happiness would be worthless is simply too ridiculous a thought to be taken seriously—given that anything at all is to be regarded valuable, an assumption I concluded valid in the beginning of this post.