I suspect your fridge logic would be solved by fvzcyl abg trggvat qb jung ur jnagrq, hagvy ur jvfurq ng fbzr cbvag gung ur jbhyq abg or n fbpvbcngu. I’m more worried about the part you rot13ed, and I suspect it’s part of what makes Eliezer consider it horror. I feel that’s the main horror part of the story.
There are also the issues of Celestia lying to Lavendar when clearly she wants the truth on some level, the worry about those who would have uploaded (or uploaded earlier) if they had a human option, and the lack of obviously-possible medical and other care for the unuploaded humans (whose values could be satisfied almost as much as those of the ponies). These are instances when an AI is almost-but-not-quite Friendly (and, in the case of the simple fictional story instead of everyday life, could have been easily avoided by telling Celestia to “satisfy values” and that most people she meets initially want friendship and ponies). These are probably the parts that Eliezer is referring to, because of his work in avoiding uFAI and almost-FAI. On the other hand, they are far better than his default scenario, the no AI scenario, and the Failed Utopia #4-2 scenario in the OP. EDIT: Additionally, in the story at least, everything except the lying was easily avoidable by having Celestia just maximize values, while telling her that most people she meets early on will value friendship and ponies (and the lying at the end seems to be somewhat out-of-character because it doesn’t actually maximize values).
One other thing some might find horrifying, but probably not Eliezer, is the “Does Síofra die” question. To me, and I presume to him, the answer is “surely not”, and the question of ethics boils down to a simple check “does there ever exist an observer moment without a successor; i.e., has somebody died?”. Obviously some people do die preventable deaths, but Síofra isn’t one of them.
I suspect your fridge logic would be solved by fvzcyl abg trggvat qb jung ur jnagrq, hagvy ur jvfurq ng fbzr cbvag gung ur jbhyq abg or n fbpvbcngu. I’m more worried about the part you rot13ed, and I suspect it’s part of what makes Eliezer consider it horror. I feel that’s the main horror part of the story.
There are also the issues of Celestia lying to Lavendar when clearly she wants the truth on some level, the worry about those who would have uploaded (or uploaded earlier) if they had a human option, and the lack of obviously-possible medical and other care for the unuploaded humans (whose values could be satisfied almost as much as those of the ponies). These are instances when an AI is almost-but-not-quite Friendly (and, in the case of the simple fictional story instead of everyday life, could have been easily avoided by telling Celestia to “satisfy values” and that most people she meets initially want friendship and ponies). These are probably the parts that Eliezer is referring to, because of his work in avoiding uFAI and almost-FAI. On the other hand, they are far better than his default scenario, the no AI scenario, and the Failed Utopia #4-2 scenario in the OP. EDIT: Additionally, in the story at least, everything except the lying was easily avoidable by having Celestia just maximize values, while telling her that most people she meets early on will value friendship and ponies (and the lying at the end seems to be somewhat out-of-character because it doesn’t actually maximize values).
One other thing some might find horrifying, but probably not Eliezer, is the “Does Síofra die” question. To me, and I presume to him, the answer is “surely not”, and the question of ethics boils down to a simple check “does there ever exist an observer moment without a successor; i.e., has somebody died?”. Obviously some people do die preventable deaths, but Síofra isn’t one of them.