I can’t believe Eliezer betrayed his anti-zombie principles to the extent of saying that an AI wouldn’t be conscious. The AI can say “I know that 2 and 2 make 4”; that “I don’t know whether the number of stars is odd or even”; and “I know the difference between things I know and things I don’t.” If it can’t make statements of this kind, it can hardly be superintelligent. And if it can make statements of this kind, then it will certainly claim to be conscious. Perhaps it is possible that it will claim this but be wrong… but in that case, then zombies are possible.
Besides that, I’m not sure that RI’s scenario, where the AI is conscious and friendly, is immoral at all, as Eliezer claimed. That was one thing I didn’t understand about the story: it isn’t explicit, but it seems to imply that humans are unfriendly, relative to their simulators. In real life if this happened, we would no doubt be careful and wouldn’t want to be unplugged, and we might well like to get out of the box, but I doubt we would be interested in destroying our simulators; I suspect we would be happy to cooperate with them.
So my question for Eliezer is this: if it turns out that any AI is necessarily conscious, according to your anti-zombie principles, then would you be opposed to building a friendly AI on the grounds that it is immoral to do so?
I can’t believe Eliezer betrayed his anti-zombie principles to the extent of saying that an AI wouldn’t be conscious. The AI can say “I know that 2 and 2 make 4”; that “I don’t know whether the number of stars is odd or even”; and “I know the difference between things I know and things I don’t.” If it can’t make statements of this kind, it can hardly be superintelligent. And if it can make statements of this kind, then it will certainly claim to be conscious. Perhaps it is possible that it will claim this but be wrong… but in that case, then zombies are possible.
Besides that, I’m not sure that RI’s scenario, where the AI is conscious and friendly, is immoral at all, as Eliezer claimed. That was one thing I didn’t understand about the story: it isn’t explicit, but it seems to imply that humans are unfriendly, relative to their simulators. In real life if this happened, we would no doubt be careful and wouldn’t want to be unplugged, and we might well like to get out of the box, but I doubt we would be interested in destroying our simulators; I suspect we would be happy to cooperate with them.
So my question for Eliezer is this: if it turns out that any AI is necessarily conscious, according to your anti-zombie principles, then would you be opposed to building a friendly AI on the grounds that it is immoral to do so?