In Bostrom’s formulation of Pascal’s mugging, Pascal incorrectly limits the possibilities to two:
The mugger just runs off with his money. (High probability, small negative utility.)
The mugger really is a benevolent magic being, and blesses Pascal with 1,000 quadrillion years of additional happy life. (Very low probability, very big positive utility.)
But Pascal is wrong to ignore the third possibility that the mugger really is a magic being, but a malevolent one, who will curse Pascal with 1,000 quadrillion years of torture and then kill him. (Very low probability, very big negative utility.)
The mugger doesn’t mention this possibility, but Pascal is mistaken to not consider it.
Pascal’s credence in the mugger’s malice and deceit should be at least as strong as his credence in the mugger’s benevolence and truthfulness. And so, this possibility cancels out the positive expected utility from the possibility that the mugger does mention.
There is a large space of such fantasy possibilities, all of which are about as likely as the mugger’s claim. It is a mistake to privilege one of them (benevolent magic being) over all the countless others.
There are also plenty that are much more likely, such as “the mugger uses Pascal’s money to go buy a gun, then comes back and robs Pascal’s house too, because why rob a sucker once when you can rob him twice (and lay the blame on him for enabling you to do it)?”
It’s still a mistake to privilege a particular fantasy mugger god story over all other fantasy mugger god stories.
You are being acausally-mugged in every direction, all at once, all the time, forever. If one FMG tells you to do action A right now, well, if you did that, you’d be disregarding all other FMGs that tell you to do B, C, D, etc. right now. You cannot possibly comply with all the myriad demands of all possible FMGs; you certainly can’t do so proportional to those FMGs’ chance of realness; nor can you a-priori discern which FMGs are realer than others with sufficient precision to generate an optimal course of action. The space of FMGs is too big and the mapping to their preferred actions is too intractable.
(And no, I’m not sure we can even discount FMGs who would, if created, regret their own existence. They might well be outnumbered by FMGs who want to exist — but perhaps their preference for nonexisting is much, much stronger. Some FMGs are miserable bastards, like AM in “I Have No Mouth And I Must Scream”. Please don’t build one.)
It’s not just combinatorial explosion; it’s also chaos. How do you get an FMG? Write a blog-post story of a god; figure out what that god would want you to do; then do that. But two stories that are nearby in story-space can generate action recommendations that are wildly different or even opposed. The parts of FMG-space that deviate from conventional ethics & epistemology offer no guidance because they diverge into chaos.
No, decision theories just don’t give us free a-priori perfect knowledge of the precise will of a vengeful & intolerant god we just made up for a story. They’re still fine for real world situations like keeping your promises to other people.
In Bostrom’s formulation of Pascal’s mugging, Pascal incorrectly limits the possibilities to two:
The mugger just runs off with his money. (High probability, small negative utility.)
The mugger really is a benevolent magic being, and blesses Pascal with 1,000 quadrillion years of additional happy life. (Very low probability, very big positive utility.)
But Pascal is wrong to ignore the third possibility that the mugger really is a magic being, but a malevolent one, who will curse Pascal with 1,000 quadrillion years of torture and then kill him. (Very low probability, very big negative utility.)
The mugger doesn’t mention this possibility, but Pascal is mistaken to not consider it.
Pascal’s credence in the mugger’s malice and deceit should be at least as strong as his credence in the mugger’s benevolence and truthfulness. And so, this possibility cancels out the positive expected utility from the possibility that the mugger does mention.
There is a large space of such fantasy possibilities, all of which are about as likely as the mugger’s claim. It is a mistake to privilege one of them (benevolent magic being) over all the countless others.
There are also plenty that are much more likely, such as “the mugger uses Pascal’s money to go buy a gun, then comes back and robs Pascal’s house too, because why rob a sucker once when you can rob him twice (and lay the blame on him for enabling you to do it)?”
Comment withdrawn.
It’s still a mistake to privilege a particular fantasy mugger god story over all other fantasy mugger god stories.
You are being acausally-mugged in every direction, all at once, all the time, forever. If one FMG tells you to do action A right now, well, if you did that, you’d be disregarding all other FMGs that tell you to do B, C, D, etc. right now. You cannot possibly comply with all the myriad demands of all possible FMGs; you certainly can’t do so proportional to those FMGs’ chance of realness; nor can you a-priori discern which FMGs are realer than others with sufficient precision to generate an optimal course of action. The space of FMGs is too big and the mapping to their preferred actions is too intractable.
(And no, I’m not sure we can even discount FMGs who would, if created, regret their own existence. They might well be outnumbered by FMGs who want to exist — but perhaps their preference for nonexisting is much, much stronger. Some FMGs are miserable bastards, like AM in “I Have No Mouth And I Must Scream”. Please don’t build one.)
◉
It’s not just combinatorial explosion; it’s also chaos. How do you get an FMG? Write a blog-post story of a god; figure out what that god would want you to do; then do that. But two stories that are nearby in story-space can generate action recommendations that are wildly different or even opposed. The parts of FMG-space that deviate from conventional ethics & epistemology offer no guidance because they diverge into chaos.
Comment withdrawn.
No, decision theories just don’t give us free a-priori perfect knowledge of the precise will of a vengeful & intolerant god we just made up for a story. They’re still fine for real world situations like keeping your promises to other people.
Comment withdrawn.