Pascal’s Mugging: Tiny Probabilities of Vast Utilities

The most com­mon for­mal­iza­tions of Oc­cam’s Ra­zor, Solomonoff in­duc­tion and Min­i­mum De­scrip­tion Length, mea­sure the pro­gram size of a com­pu­ta­tion used in a hy­poth­e­sis, but don’t mea­sure the run­ning time or space re­quire­ments of the com­pu­ta­tion. What if this makes a mind vuln­er­a­ble to finite forms of Pas­cal’s Wager? A com­pactly speci­fied wa­ger can grow in size much faster than it grows in com­plex­ity. The util­ity of a Tur­ing ma­chine can grow much faster than its prior prob­a­bil­ity shrinks.

Con­sider Knuth’s up-ar­row no­ta­tion:

  • 3^3 = 3*3*3 = 27

  • 3^^3 = (3^(3^3)) = 3^27 = 3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3*3 = 7625597484987

  • 3^^^3 = (3^^(3^^3)) = 3^^7625597484987 = 3^(3^(3^(… 7625597484987 times …)))

In other words: 3^^^3 de­scribes an ex­po­nen­tial tower of threes 7625597484987 lay­ers tall. Since this num­ber can be com­puted by a sim­ple Tur­ing ma­chine, it con­tains very lit­tle in­for­ma­tion and re­quires a very short mes­sage to de­scribe. This, even though writ­ing out 3^^^3 in base 10 would re­quire enor­mously more writ­ing ma­te­rial than there are atoms in the known uni­verse (a paltry 10^80).

Now sup­pose some­one comes to me and says, “Give me five dol­lars, or I’ll use my magic pow­ers from out­side the Ma­trix to run a Tur­ing ma­chine that simu­lates and kills 3^^^^3 peo­ple.”

Call this Pas­cal’s Mug­ging.

“Magic pow­ers from out­side the Ma­trix” are eas­ier said than done—we have to sup­pose that our world is a com­put­ing simu­la­tion run from within an en­vi­ron­ment that can af­ford simu­la­tion of ar­bi­trar­ily large finite Tur­ing ma­chines, and that the would-be wiz­ard has been spliced into our own Tur­ing tape and is in con­tin­u­ing com­mu­ni­ca­tion with an out­side op­er­a­tor, etc.

Thus the Kol­mogorov com­plex­ity of “magic pow­ers from out­side the Ma­trix” is larger than the mere English words would in­di­cate. There­fore the Solomonoff-in­ducted prob­a­bil­ity, two to the nega­tive Kol­mogorov com­plex­ity, is ex­po­nen­tially tinier than one might naively think.

But, small as this prob­a­bil­ity is, it isn’t any­where near as small as 3^^^^3 is large. If you take a dec­i­mal point, fol­lowed by a num­ber of ze­ros equal to the length of the Bible, fol­lowed by a 1, and mul­ti­ply this uni­mag­in­ably tiny frac­tion by 3^^^^3, the re­sult is pretty much 3^^^^3.

Most peo­ple, I think, en­vi­sion an “in­finite” God that is nowhere near as large as 3^^^^3. “In­finity” is re­as­sur­ingly fea­ture­less and blank. “Eter­nal life in Heaven” is nowhere near as in­timi­dat­ing as the thought of spend­ing 3^^^^3 years on one of those fluffy clouds. The no­tion that the di­ver­sity of life on Earth springs from God’s in­finite cre­ativity, sounds more plau­si­ble than the no­tion that life on Earth was cre­ated by a su­per­in­tel­li­gence 3^^^^3 bits large. Similarly for en­vi­sion­ing an “in­finite” God in­ter­ested in whether women wear men’s cloth­ing, ver­sus a su­per­in­tel­li­gence of 3^^^^3 bits, etc.

The origi­nal ver­sion of Pas­cal’s Wager is eas­ily dealt with by the gi­gan­tic mul­ti­plic­ity of pos­si­ble gods, an Allah for ev­ery Christ and a Zeus for ev­ery Allah, in­clud­ing the “Pro­fes­sor God” who places only athe­ists in Heaven. And since all the ex­pected util­ities here are allegedly “in­finite”, it’s easy enough to ar­gue that they can­cel out. In­fini­ties, be­ing fea­ture­less and blank, are all the same size.

But sup­pose I built an AI which worked by some bounded analogue of Solomonoff in­duc­tion—an AI suffi­ciently Bayesian to in­sist on calcu­lat­ing com­plex­ities and as­sess­ing prob­a­bil­ities, rather than just wav­ing them off as “large” or “small”.

If the prob­a­bil­ities of var­i­ous sce­nar­ios con­sid­ered did not ex­actly can­cel out, the AI’s ac­tion in the case of Pas­cal’s Mug­ging would be over­whelm­ingly dom­i­nated by what­ever tiny differ­en­tials ex­isted in the var­i­ous tiny prob­a­bil­ities un­der which 3^^^^3 units of ex­pected util­ity were ac­tu­ally at stake.

You or I would prob­a­bly wave off the whole mat­ter with a laugh, plan­ning ac­cord­ing to the dom­i­nant main­line prob­a­bil­ity: Pas­cal’s Mug­ger is just a philoso­pher out for a fast buck.

But a sili­con chip does not look over the code fed to it, as­sess it for rea­son­able­ness, and cor­rect it if not. An AI is not given its code like a hu­man ser­vant given in­struc­tions. An AI is its code. What if a philoso­pher tries Pas­cal’s Mug­ging on the AI for a joke, and the tiny prob­a­bil­ities of 3^^^^3 lives be­ing at stake, over­ride ev­ery­thing else in the AI’s calcu­la­tions? What is the mere Earth at stake, com­pared to a tiny prob­a­bil­ity of 3^^^^3 lives?

How do I know to be wor­ried by this line of rea­son­ing? How do I know to ra­tio­nal­ize rea­sons a Bayesian shouldn’t work that way? A mind that worked strictly by Solomonoff in­duc­tion would not know to ra­tio­nal­ize rea­sons that Pas­cal’s Mug­ging mat­tered less than Earth’s ex­is­tence. It would sim­ply go by what­ever an­swer Solomonoff in­duc­tion ob­tained.

It would seem, then, that I’ve im­plic­itly de­clared my ex­is­tence as a mind that does not work by the logic of Solomonoff, at least not the way I’ve de­scribed it. What am I com­par­ing Solomonoff’s an­swer to, to de­ter­mine whether Solomonoff in­duc­tion got it “right” or “wrong”?

Why do I think it’s un­rea­son­able to fo­cus my en­tire at­ten­tion on the magic-bear­ing pos­si­ble wor­lds, faced with a Pas­cal’s Mug­ging? Do I have an in­stinct to re­sist ex­ploita­tion by ar­gu­ments “any­one could make”? Am I un­satis­fied by any vi­su­al­iza­tion in which the dom­i­nant main­line prob­a­bil­ity leads to a loss? Do I drop suffi­ciently small prob­a­bil­ities from con­sid­er­a­tion en­tirely? Would an AI that lacks these in­stincts be ex­ploitable by Pas­cal’s Mug­ging?

Is it me who’s wrong? Should I worry more about the pos­si­bil­ity of some Unseen Mag­i­cal Prankster of very tiny prob­a­bil­ity tak­ing this post liter­ally, than about the fate of the hu­man species in the “main­line” prob­a­bil­ities?

It doesn’t feel to me like 3^^^^3 lives are re­ally at stake, even at very tiny prob­a­bil­ity. I’d sooner ques­tion my grasp of “ra­tio­nal­ity” than give five dol­lars to a Pas­cal’s Mug­ger be­cause I thought it was “ra­tio­nal”.

Should we pe­nal­ize com­pu­ta­tions with large space and time re­quire­ments? This is a hack that solves the prob­lem, but is it true? Are com­pu­ta­tion­ally costly ex­pla­na­tions less likely? Should I think the uni­verse is prob­a­bly a coarse-grained simu­la­tion of my mind rather than real quan­tum physics, be­cause a coarse-grained hu­man mind is ex­po­nen­tially cheaper than real quan­tum physics? Should I think the galax­ies are tiny lights on a painted back­drop, be­cause that Tur­ing ma­chine would re­quire less space to com­pute?

Given that, in gen­eral, a Tur­ing ma­chine can in­crease in util­ity vastly faster than it in­creases in com­plex­ity, how should an Oc­cam-abid­ing mind avoid be­ing dom­i­nated by tiny prob­a­bil­ities of vast util­ities?

If I could for­mal­ize whichever in­ter­nal crite­rion was tel­ling me I didn’t want this to hap­pen, I might have an an­swer.

I talked over a var­i­ant of this prob­lem with Nick Hay, Peter de Blanc, and Mar­cello Her­reshoff in sum­mer of 2006. I don’t feel I have a satis­fac­tory re­s­olu­tion as yet, so I’m throw­ing it open to any an­a­lytic philoso­phers who might hap­pen to read Over­com­ing Bias.