Okay, Hirou has evidence that Vhazhar is a moral savant. But the reader, and Hirou, sees little evidence that Vhazhar has worked out a formal, rigorous theory of Friendliness. I thought that anything less than that, on your view, virtually guaranteed the obliteration of almost everything valuable.
But I draw a weaker inference from Vhazhar’s ability to overcome indoctrination. Yes, it implies that he probably had a high native aptitude for correct moral reasoning. But the very fact that he was subjected to the indoctrination means that he’s probably damaged anyways. If someone survives a disease that’s usually deadly, you should expect that she went into the disease with an uncommonly strong constitution. But, given that she’s had the disease, you should expect that she’s now less healthy than average.
But the reader, and Hirou, sees little evidence that Vhazhar has worked out a formal, rigorous theory of Friendliness. I thought that anything less than that, on your view, virtually guaranteed the obliteration of almost everything valuable.
Only by AIs. Human uploads would be a whole different story. Not necessarily a good story, but a different story, and one in which—whatever the objective frequency of winning—I’d have to say that, relative to my subjective knowledge, there’s a pretty sizable chunk of chance.
If Vhazhar was literally casting a spell to run the world directly, and he wasn’t able to take advantage of moral magic like that embodied in the Sword of Good itself (which, conceivably, could be a lot less sophisticated than its name implies) then it’s a full-fledged Friendly AI problem.
What are the justifiable expectations one could have about the Sword of Good? In particular, why suppose that it’s a Sword of Good in anything other than name only? Why suppose that it’s any protection against evil?
I also didn’t consider the possibility that Vhazhar was planning to run the world himself directly. A human just doesn’t have the computational capacity to run the world. If a human tried to run the world, there would still be both fortune and misfortune.
For that reason, I assumed that his plan was for some extrapolated version of his volition to run the world. But if he’s created something that will implement his CEV accurately, hasn’t he solved FAI?
I also didn’t consider the possibility that Vhazhar was planning to run the world himself directly. A human just doesn’t have the computational capacity to run the world. If a human tried to run the world, there would still be both fortune and misfortune.
There could be less misfortune. A cautious human god who wasn’t corrupted by power certainly could plausibly accomplish a lot of good with a few minimal actions. Of course the shaky part is that “cautious” and “not corrupted” part.
Where does the ability to specify complex wishes become distinct from the ability to implement them though? What are the capabilities of a god with human mind? If there is a lot of automation for implementing the wishes, how much of the person’s preference does this automation anticipate? In what sense does the limitation on a god’s mind to be merely human affect god’s capacity to control the world? There doesn’t seem to be a natural concept that captures this.
Okay. I had taken the Prophecy of Doom to be saying that there would no longer be both “luck and misfortune”. I can see that it could be read otherwise, though.
Well, there are at least several obvious fixes that we humans would want to make to the world we live in, but are unable to. For example, we would like to wipe out the malaria parasite that infects humans. The dragon is bad, the world is full of really, really horrible things, and I’d rather just make it stop rather than worry too much about being corrupted by power.
Okay, Hirou has evidence that Vhazhar is a moral savant. But the reader, and Hirou, sees little evidence that Vhazhar has worked out a formal, rigorous theory of Friendliness. I thought that anything less than that, on your view, virtually guaranteed the obliteration of almost everything valuable.
But I draw a weaker inference from Vhazhar’s ability to overcome indoctrination. Yes, it implies that he probably had a high native aptitude for correct moral reasoning. But the very fact that he was subjected to the indoctrination means that he’s probably damaged anyways. If someone survives a disease that’s usually deadly, you should expect that she went into the disease with an uncommonly strong constitution. But, given that she’s had the disease, you should expect that she’s now less healthy than average.
Only by AIs. Human uploads would be a whole different story. Not necessarily a good story, but a different story, and one in which—whatever the objective frequency of winning—I’d have to say that, relative to my subjective knowledge, there’s a pretty sizable chunk of chance.
If Vhazhar was literally casting a spell to run the world directly, and he wasn’t able to take advantage of moral magic like that embodied in the Sword of Good itself (which, conceivably, could be a lot less sophisticated than its name implies) then it’s a full-fledged Friendly AI problem.
What are the justifiable expectations one could have about the Sword of Good? In particular, why suppose that it’s a Sword of Good in anything other than name only? Why suppose that it’s any protection against evil?
I also didn’t consider the possibility that Vhazhar was planning to run the world himself directly. A human just doesn’t have the computational capacity to run the world. If a human tried to run the world, there would still be both fortune and misfortune.
For that reason, I assumed that his plan was for some extrapolated version of his volition to run the world. But if he’s created something that will implement his CEV accurately, hasn’t he solved FAI?
There could be less misfortune. A cautious human god who wasn’t corrupted by power certainly could plausibly accomplish a lot of good with a few minimal actions. Of course the shaky part is that “cautious” and “not corrupted” part.
Where does the ability to specify complex wishes become distinct from the ability to implement them though? What are the capabilities of a god with human mind? If there is a lot of automation for implementing the wishes, how much of the person’s preference does this automation anticipate? In what sense does the limitation on a god’s mind to be merely human affect god’s capacity to control the world? There doesn’t seem to be a natural concept that captures this.
Okay. I had taken the Prophecy of Doom to be saying that there would no longer be both “luck and misfortune”. I can see that it could be read otherwise, though.
Well, there are at least several obvious fixes that we humans would want to make to the world we live in, but are unable to. For example, we would like to wipe out the malaria parasite that infects humans. The dragon is bad, the world is full of really, really horrible things, and I’d rather just make it stop rather than worry too much about being corrupted by power.