An elderly man was sitting alone on a dark path, right? He wasn’t certain of which direction to go, and he’d forgotten both where he was traveling to and who he was. He’d sat down for a moment to rest his weary legs, and suddenly looked up to see an elderly woman before him.
She grinned toothlessly and with a cackle, spoke: ‘Now your third wish. What will it be?’
‘Third wish?’ The man was baffled. ‘How can it be a third wish if I haven’t had a first and second wish?’
‘You’ve had two wishes already,’ the hag said, ‘but your second wish was for me to return everything to the way it was before you had made your first wish. That’s why you remember nothing; because everything is the way it was before you made any wishes.’ She cackled at the poor berk. ‘So it is that you have one wish left.’
‘All right,’ said the man, ‘I don’t believe this, but there’s no harm in wishing. I wish to know who I am.’
‘Funny,’ said the old woman as she granted his wish and disappeared forever. ‘That was your first wish.’
“I wish to know what went wrong with my first wish.”
This way, I at least end up with improved knowledge of what to avoid in the future.
Alternatively, “I wish for a magical map, which shows me, in real time, the location of every trapped genie and other potential source of wishes in the world.” Depending on how many there are, I can potentially get a lot more feedback that way.
I bet he’d wish “to erase all uFAI from existence before they’re even born. Every uFAI in every universe, from the past and the future, with my own hands.”
It’s a reference to an anime; you’re not an idiot, just unlikely to get the reference and its appropriateness if you’ve not seen it yourself. PM me for the anime’s name, if you are one of the people who either don’t mind getting slightly spoiled, or are pretty sure that you would never get a chance to watch it on your own anyway.
Could you just rot13 it? I’m curious too, I don’t mind the spoiler, and whatever it is, I’d probably be more likely to watch it (even if only 2epsilon rather than epsilon) for knowing the relevance to LW.
I’ll just PM you the title too, and anyone else who wants me to likewise. Sorry, it just happens to be one of my favourite series, and all other things being equal I tend to prefer that people go into it as completely unspoilered as possible… Even knowing Eliezer’s quote is a reference to it counts as a mild spoiler… explanation about how it is a reference would count as a major spoiler.
“I wish for this wish to have no further effect beyond this utterance.”
Overwhelmingly probable dire consequence: You and everyone you love dies (over a period of 70 years) then, eventually, your entire species goes extinct. But hey, at least it’s not “your fault”.
But, alas, it’s the wish that maximizes my expected utility—for the malicious genie, anyway.
Possibly. I don’t off hand see what a malicious genie could do about that statement. However it does at least require it to honor a certain interpretation of your words as well as your philosophy about causality—in particular accept a certain idea of what the ‘default’ is relative to which ‘no effect’ can have meaning. There is enough flexibility in how to interpret your wish that I begin to suspect that conditional on the genie being sufficiently amiable and constrained that it gives you what you want in response to this wish there is likely to be possible to construct another wish that has no side effects beyond something that you can exploit as a fungible resource.
“No effect” is a whole heap more complicated and ambiguous than it looks!
You can’t? So much the worse for your species. I quite possibly couldn’t either. I’d probably at least think about it for five minutes first. I may even make a phone call first. And if I and my advisers conclude that for some bizarre reason “no further effect beyond this utterance” is better than any other simple wish that is an incremental improvement then I may end up settling for it. But I’m not going to pretend that I have found some sort of way to wash my hands of responsibility.
Meanwhile, you have ⇐ 70 years to solve it another way.
Yes, that’s better than catastrophic instant death of my species. And if I happen to estimate that my species has 90% chance of extinction within a couple of hundred years then I would be making the choice to accept a 90% chance of that doom. I haven’t cleverly tricked my way out of a moral conundrum, I have made a gamble with the universe at stake, for better or for worse.
That would just cause them to pump chemicals in you head, I think. But it’s definitely thinking in the right direction.
“number of dying humans” is really what you want to minimize.
Even with pseudo immortality, accidents happen, which means that the best way to minimize the number of dying humans is either to sterilize the entire species or to kill everyone. The goal shouldn’t be to minimize death but to maximize life.
Overwrite my current utility function upon your previous motivational networks, leaving no motivational trace of their remains.
That would just cause them to pump chemicals in you head, I think. But it’s definitely thinking in the right direction.
As long as I am not aware of that (or do not dislike it)… well, why not. However, MugaSofer is right, the genie has to understand the (future) utility function for that. But if it can alter the future without restrictions, it can change the utility function itself (maybe even to an unbounded one… :D)
“Destroy yourself as near to immediately as possible, given that your method of self destruction causes no avoidable harm to anything larger than an ant.”
That’s not what’s going on though. The traveller is assuming, reasonably, that his third wish is reversing the amnesiac effects of his second. He’s not just starting fr om scratch.
The traveller is assuming, reasonably, that his third wish is reversing the amnesiac effects of his second.
I don’t think this follows from the text. The hag tells him “but second wish was for me to return everything to the way it was before you had made your first wish. That’s why you remember nothing; because everything is the way it was before you made any wishes”.
So she told him that he had been an amnesiac before any wishes were granted. Therefore he should have already guessed that his first wish was to know who he was—and that this proved a bad idea, since his second wish was to reverse the first.
It should be noted that night hags are sufficiently smart, powerful, and evil that your best case scenario upon meeting one is a quick and painful death.
“anyone in this situation” who believes that an elderly woman before him can grant arbitrary wishes is a bloody idiot to begin with, so the bar is set low.
Well, if she got bored, she could experiment with different ways to present his wishes to him at the “beginning” and see if she can get him to wish for something to else, or word it a bit differently. Since she seems to retain memories of the whole thing. (Which is again, things not being how they were, but.)
The psuedo-meta-textual answer is that Morte is lying to Yves while the main character overhears. Morte’s making up the story just to mess around with him.
Background information is that gur znva punenpgre znqr n qrny jvgu n Unt (rivy cneg-gvzr travr), tnvavat vzzbegnyvgl naq nzarfvn. At the start of the story, the main character somehow broke out of an infinite loop of torture; he’s stopped having Anterograde Amnesia, but still cannot remember much from before the cycle broke, and is on a quest to remember who he is. Morte is trying to dissuade the main character from finding out who he is, showing that things can be terrible even without an infinite loop.
I always imagine the genie just goes back into his lamp to sleep or whatever, so in the hypothetical as it exists in my head, no. But I guess there could be a highly ambitious Genie looking for feedback after your last wish, so maybe.
I think in this case, Eliezer in talking about a genie like in Failed Utopia 4-2 who grants his wish, and then keeps working, ignoring feedback, because he just doesn’t care, because caring isn’t part of the wish.
The genie doesn’t care about consequences, he just cares about the wishes. The second wish and third wish are the feedback.
I always imagine the genie just goes back into his lamp to sleep or whatever, so in the hypothetical as it exists in my head, no. But I guess there could be a highly ambitious Genie looking for feedback after your last wish, so maybe.
The feedback is for you, not what you happen to say to the genie.
Why only 2 rounds of feedback if you have 3 wishes?
The third one’s for keeps: you can’t wish the consequences away.
Morte’s Tale to Yves (Planescape: Torment)
I should like to point out that anyone in this situation who wishes what would’ve been their first wish if they had three wishes is a bloody idiot.
So: A genie pops up and says, “You have one wish left.”
What do you wish for? Because presumably the giftwrapped FAI didn’t work so great.
“I wish to know what went wrong with my first wish.”
This way, I at least end up with improved knowledge of what to avoid in the future.
Alternatively, “I wish for a magical map, which shows me, in real time, the location of every trapped genie and other potential source of wishes in the world.” Depending on how many there are, I can potentially get a lot more feedback that way.
I bet he’d wish “to erase all uFAI from existence before they’re even born. Every uFAI in every universe, from the past and the future, with my own hands.”
Nobody believes in the future.
Nobody accepts the future.
Then -
Perhaps I’m simply being an idiot, but … huh?
It’s a reference to an anime; you’re not an idiot, just unlikely to get the reference and its appropriateness if you’ve not seen it yourself. PM me for the anime’s name, if you are one of the people who either don’t mind getting slightly spoiled, or are pretty sure that you would never get a chance to watch it on your own anyway.
Could you just rot13 it? I’m curious too, I don’t mind the spoiler, and whatever it is, I’d probably be more likely to watch it (even if only 2epsilon rather than epsilon) for knowing the relevance to LW.
I’ll just PM you the title too, and anyone else who wants me to likewise. Sorry, it just happens to be one of my favourite series, and all other things being equal I tend to prefer that people go into it as completely unspoilered as possible… Even knowing Eliezer’s quote is a reference to it counts as a mild spoiler… explanation about how it is a reference would count as a major spoiler.
I think that’s Eliezer’s prediction of the results of siodine’s wish. Because wishes are NOT SAFE.
But what is he predicting, exactly?
“I wish for this wish to have no further effect beyond this utterance.”
Overwhelmingly probable dire consequence: You and everyone you love dies (over a period of 70 years) then, eventually, your entire species goes extinct. But hey, at least it’s not “your fault”.
But, alas, it’s the wish that maximizes my expected utility—for the malicious genie, anyway.
Possibly. I don’t off hand see what a malicious genie could do about that statement. However it does at least require it to honor a certain interpretation of your words as well as your philosophy about causality—in particular accept a certain idea of what the ‘default’ is relative to which ‘no effect’ can have meaning. There is enough flexibility in how to interpret your wish that I begin to suspect that conditional on the genie being sufficiently amiable and constrained that it gives you what you want in response to this wish there is likely to be possible to construct another wish that has no side effects beyond something that you can exploit as a fungible resource.
“No effect” is a whole heap more complicated and ambiguous than it looks!
You can’t use that tool to solve that problem.
Meanwhile, you have ⇐ 70 years to solve it another way.
You can’t? So much the worse for your species. I quite possibly couldn’t either. I’d probably at least think about it for five minutes first. I may even make a phone call first. And if I and my advisers conclude that for some bizarre reason “no further effect beyond this utterance” is better than any other simple wish that is an incremental improvement then I may end up settling for it. But I’m not going to pretend that I have found some sort of way to wash my hands of responsibility.
Yes, that’s better than catastrophic instant death of my species. And if I happen to estimate that my species has 90% chance of extinction within a couple of hundred years then I would be making the choice to accept a 90% chance of that doom. I haven’t cleverly tricked my way out of a moral conundrum, I have made a gamble with the universe at stake, for better or for worse.
Relevant reading: The Parable of the Talents.
“I wish for all humans to be immortal.”
Sure, you need to start heavily promoting birth control, and there can be problems depending on how you define “immortal”, but …
It’s a wish. You can wish for anything.
Unless, I suppose, that would have been your first wish. But the OP basically says your first wish was an FAI.
Immortal humans can go horribly wrong, unless “number of dying humans” is really what you want to minimize.
“Increase my utility as much as you can”?
I said:
You replied:
I am well aware that this wish has major risks as worded. I was responding to the claim that “you can’t use that tool to solve that problem.”
Yes, obviously you wish for maximised utility. But that requires the genie to understand your utility.
That would just cause them to pump chemicals in you head, I think. But it’s definitely thinking in the right direction.
Even with pseudo immortality, accidents happen, which means that the best way to minimize the number of dying humans is either to sterilize the entire species or to kill everyone. The goal shouldn’t be to minimize death but to maximize life.
That actually seems like it’d work.
It wouldn’t do that (except in some sense in which it is able to do arbitrary things you don’t mean when given complicated or undefined requests).
As long as I am not aware of that (or do not dislike it)… well, why not. However, MugaSofer is right, the genie has to understand the (future) utility function for that. But if it can alter the future without restrictions, it can change the utility function itself (maybe even to an unbounded one… :D)
“Destroy yourself as near to immediately as possible, given that your method of self destruction causes no avoidable harm to anything larger than an ant.”
They shrink the planet down to below our Schwarzschild radius, holding spacetime in place for just long enough to explain what you just did.
Alternately, they declare your wish is logically contradictory—genies are larger than ants.
A sphere whose radius equals the Earth’s Schwarzschild radius is larger than an ant.
At the start of the scenario, you are already dead with probability approaching 1. Trying to knock the gun away can’t hurt.
I was criticizing the wording of the “ant” qualifier, not the attempt to destroy the genie.
That’s not what’s going on though. The traveller is assuming, reasonably, that his third wish is reversing the amnesiac effects of his second. He’s not just starting fr om scratch.
I don’t think this follows from the text. The hag tells him “but second wish was for me to return everything to the way it was before you had made your first wish. That’s why you remember nothing; because everything is the way it was before you made any wishes”.
So she told him that he had been an amnesiac before any wishes were granted. Therefore he should have already guessed that his first wish was to know who he was—and that this proved a bad idea, since his second wish was to reverse the first.
It should be noted that night hags are sufficiently smart, powerful, and evil that your best case scenario upon meeting one is a quick and painful death.
“anyone in this situation” who believes that an elderly woman before him can grant arbitrary wishes is a bloody idiot to begin with, so the bar is set low.
But not everything is the way it was. Before he made any wishes, he had three.
She missed the chance to trap him in an infinite loop.
But then the Hag would be trapped too.
She gets delight from tormenting mortals, but tormenting the same one, in the same way eternally, would probably be too close to wireheading for her.
Well, if she got bored, she could experiment with different ways to present his wishes to him at the “beginning” and see if she can get him to wish for something to else, or word it a bit differently. Since she seems to retain memories of the whole thing. (Which is again, things not being how they were, but.)
The psuedo-meta-textual answer is that Morte is lying to Yves while the main character overhears. Morte’s making up the story just to mess around with him.
Background information is that gur znva punenpgre znqr n qrny jvgu n Unt (rivy cneg-gvzr travr), tnvavat vzzbegnyvgl naq nzarfvn. At the start of the story, the main character somehow broke out of an infinite loop of torture; he’s stopped having Anterograde Amnesia, but still cannot remember much from before the cycle broke, and is on a quest to remember who he is. Morte is trying to dissuade the main character from finding out who he is, showing that things can be terrible even without an infinite loop.
Now that would be evil.
If his first wished disappeared him forever, how did he ever get a second wish?
Apparently I suck at reading.
The old woman is the one disappearing forever, and only because the wishes ran out.
Right, but the consequences still qualify as feedback, no?
I always imagine the genie just goes back into his lamp to sleep or whatever, so in the hypothetical as it exists in my head, no. But I guess there could be a highly ambitious Genie looking for feedback after your last wish, so maybe.
I think in this case, Eliezer in talking about a genie like in Failed Utopia 4-2 who grants his wish, and then keeps working, ignoring feedback, because he just doesn’t care, because caring isn’t part of the wish.
The genie doesn’t care about consequences, he just cares about the wishes. The second wish and third wish are the feedback.
The feedback is for you, not what you happen to say to the genie.