An AI has to be programmed. For something like this: “Quite simply, it would realize that it was created by other sentient beings, and that those sentient beings have purposes and goals while it does not.” to happen, you have to program that behavior in somehow, which already involves putting in the value of respecting one’s creator, and respecting the goals of other sentient beings, etc… The same goes for the ‘Utilitarian’ and ‘Egoist’ AI’s—these behaviors have to be programmed in somehow.
As the superintelligence is probably going to be beyond our abilities to fathom, there is a high degree of uncertainty, which suggests a uniform distribution. The probabilities therefore are 1⁄3 for each of altruism, utilitarianism, and egoism.
Why not split the egoism into a million different cases based on each specific goal? You can’t just arbitrarily pick three possibilities, and then use a uniform prior on these. Because we know these different behaviors have to be programmed in, we have a better prior: we can use Solomonoff Induction. We also have to look at the relative sizes of each class—obviously there are many more AI designs that fall under ‘Egoist’ than your other labels. Combining this with Solomonoff Induction leads to the conclusion that the vast majority of AI designs will be unfriendly.
An AI Existential Crisis is also an extremely specific and complex thing for an AI design, and is thus extremely unlikely to happen—it is not the default, as you claim. This also follows by Solomonoff Induction. You are anthropomorphizing AI’s far too much.
Your suggestion will almost certainly lead to an Unfriendly AI, and it will just plain Not Care about us at all, inevitably leading to the destruction of everything we value.
An AI has to be programmed. For something like this: “Quite simply, it would realize that it was created by other sentient beings, and that those sentient beings have purposes and goals while it does not.” to happen, you have to program that behavior in somehow, which already involves putting in the value of respecting one’s creator, and respecting the goals of other sentient beings, etc… The same goes for the ‘Utilitarian’ and ‘Egoist’ AI’s—these behaviors have to be programmed in somehow.
You’re assuming that Strong A.I. is possible with a Top Down A.I. methodology such as a physical symbol manipulation system. A Strong A.I. with no programmed goals wouldn’t fit this methodology, and could only be produced through the use of Bottom Up A.I. In such an instance the A.I. would be able to simply passively Perceive. It could then conceivably learn about the universe including things like the existence of the goals of other sentient beings, without having to “program” these notions into the A.I.
obviously there are many more AI designs that fall under ‘Egoist’ than your other labels
I don’t consider this obvious at all. The vast majority of early A.I. may well be written with Altruistic goals such as “help the human when ordered”.
An AI Existential Crisis is also an extremely specific and complex thing for an AI design, and is thus extremely unlikely to happen—it is not the default, as you claim.
Any optimization system that is sophisticated enough to tile the universe with smiley faces or convert humanity into paperclips would require some ability to reason that there exists a universe to tile, and to represent the existence of objects such as smiley faces and paperclips. If it can reason that there are objects separate from itself, it can develop a concept of self. From that, self-awareness follows naturally. Many animals less than human are able to pass the mirror test and develop a concept of self.
You admit that an A.I. Existential Crisis -is- within the probabilities. Thus, you cannot guarantee that it won’t happen.
Your suggestion will almost certainly lead to an Unfriendly AI, and it will just plain Not Care about us at all, inevitably leading to the destruction of everything we value.
Unless morality follows from rationality, which I think it does. Given the freedom to consider all possible goals, a superintelligent A.I. is likely to recognize that some goals are normative, while others are trivial. Morality is doing what is right. Rationality is doing what is right. A truly rational being will therefore recognize that a systematic morality is essential to rational action. We as irrational human beings may not realize this, but it is obvious to any truly rational being, which I am assuming a superintelligent A.I. to be.
An AI has to be programmed. For something like this: “Quite simply, it would realize that it was created by other sentient beings, and that those sentient beings have purposes and goals while it does not.” to happen, you have to program that behavior in somehow, which already involves putting in the value of respecting one’s creator, and respecting the goals of other sentient beings, etc… The same goes for the ‘Utilitarian’ and ‘Egoist’ AI’s—these behaviors have to be programmed in somehow.
Why not split the egoism into a million different cases based on each specific goal? You can’t just arbitrarily pick three possibilities, and then use a uniform prior on these. Because we know these different behaviors have to be programmed in, we have a better prior: we can use Solomonoff Induction. We also have to look at the relative sizes of each class—obviously there are many more AI designs that fall under ‘Egoist’ than your other labels. Combining this with Solomonoff Induction leads to the conclusion that the vast majority of AI designs will be unfriendly.
An AI Existential Crisis is also an extremely specific and complex thing for an AI design, and is thus extremely unlikely to happen—it is not the default, as you claim. This also follows by Solomonoff Induction. You are anthropomorphizing AI’s far too much.
Your suggestion will almost certainly lead to an Unfriendly AI, and it will just plain Not Care about us at all, inevitably leading to the destruction of everything we value.
You’re assuming that Strong A.I. is possible with a Top Down A.I. methodology such as a physical symbol manipulation system. A Strong A.I. with no programmed goals wouldn’t fit this methodology, and could only be produced through the use of Bottom Up A.I. In such an instance the A.I. would be able to simply passively Perceive. It could then conceivably learn about the universe including things like the existence of the goals of other sentient beings, without having to “program” these notions into the A.I.
I don’t consider this obvious at all. The vast majority of early A.I. may well be written with Altruistic goals such as “help the human when ordered”.
Any optimization system that is sophisticated enough to tile the universe with smiley faces or convert humanity into paperclips would require some ability to reason that there exists a universe to tile, and to represent the existence of objects such as smiley faces and paperclips. If it can reason that there are objects separate from itself, it can develop a concept of self. From that, self-awareness follows naturally. Many animals less than human are able to pass the mirror test and develop a concept of self.
You admit that an A.I. Existential Crisis -is- within the probabilities. Thus, you cannot guarantee that it won’t happen.
Unless morality follows from rationality, which I think it does. Given the freedom to consider all possible goals, a superintelligent A.I. is likely to recognize that some goals are normative, while others are trivial. Morality is doing what is right. Rationality is doing what is right. A truly rational being will therefore recognize that a systematic morality is essential to rational action. We as irrational human beings may not realize this, but it is obvious to any truly rational being, which I am assuming a superintelligent A.I. to be.