But the reader, and Hirou, sees little evidence that Vhazhar has worked out a formal, rigorous theory of Friendliness. I thought that anything less than that, on your view, virtually guaranteed the obliteration of almost everything valuable.
Only by AIs. Human uploads would be a whole different story. Not necessarily a good story, but a different story, and one in which—whatever the objective frequency of winning—I’d have to say that, relative to my subjective knowledge, there’s a pretty sizable chunk of chance.
If Vhazhar was literally casting a spell to run the world directly, and he wasn’t able to take advantage of moral magic like that embodied in the Sword of Good itself (which, conceivably, could be a lot less sophisticated than its name implies) then it’s a full-fledged Friendly AI problem.
What are the justifiable expectations one could have about the Sword of Good? In particular, why suppose that it’s a Sword of Good in anything other than name only? Why suppose that it’s any protection against evil?
I also didn’t consider the possibility that Vhazhar was planning to run the world himself directly. A human just doesn’t have the computational capacity to run the world. If a human tried to run the world, there would still be both fortune and misfortune.
For that reason, I assumed that his plan was for some extrapolated version of his volition to run the world. But if he’s created something that will implement his CEV accurately, hasn’t he solved FAI?
I also didn’t consider the possibility that Vhazhar was planning to run the world himself directly. A human just doesn’t have the computational capacity to run the world. If a human tried to run the world, there would still be both fortune and misfortune.
There could be less misfortune. A cautious human god who wasn’t corrupted by power certainly could plausibly accomplish a lot of good with a few minimal actions. Of course the shaky part is that “cautious” and “not corrupted” part.
Where does the ability to specify complex wishes become distinct from the ability to implement them though? What are the capabilities of a god with human mind? If there is a lot of automation for implementing the wishes, how much of the person’s preference does this automation anticipate? In what sense does the limitation on a god’s mind to be merely human affect god’s capacity to control the world? There doesn’t seem to be a natural concept that captures this.
Okay. I had taken the Prophecy of Doom to be saying that there would no longer be both “luck and misfortune”. I can see that it could be read otherwise, though.
Only by AIs. Human uploads would be a whole different story. Not necessarily a good story, but a different story, and one in which—whatever the objective frequency of winning—I’d have to say that, relative to my subjective knowledge, there’s a pretty sizable chunk of chance.
If Vhazhar was literally casting a spell to run the world directly, and he wasn’t able to take advantage of moral magic like that embodied in the Sword of Good itself (which, conceivably, could be a lot less sophisticated than its name implies) then it’s a full-fledged Friendly AI problem.
What are the justifiable expectations one could have about the Sword of Good? In particular, why suppose that it’s a Sword of Good in anything other than name only? Why suppose that it’s any protection against evil?
I also didn’t consider the possibility that Vhazhar was planning to run the world himself directly. A human just doesn’t have the computational capacity to run the world. If a human tried to run the world, there would still be both fortune and misfortune.
For that reason, I assumed that his plan was for some extrapolated version of his volition to run the world. But if he’s created something that will implement his CEV accurately, hasn’t he solved FAI?
There could be less misfortune. A cautious human god who wasn’t corrupted by power certainly could plausibly accomplish a lot of good with a few minimal actions. Of course the shaky part is that “cautious” and “not corrupted” part.
Where does the ability to specify complex wishes become distinct from the ability to implement them though? What are the capabilities of a god with human mind? If there is a lot of automation for implementing the wishes, how much of the person’s preference does this automation anticipate? In what sense does the limitation on a god’s mind to be merely human affect god’s capacity to control the world? There doesn’t seem to be a natural concept that captures this.
Okay. I had taken the Prophecy of Doom to be saying that there would no longer be both “luck and misfortune”. I can see that it could be read otherwise, though.