Clarifying The Malignity of the Universal Prior: The Lexical Update

[UPDATE: looks like the lex­i­cal up­date is real af­ter all; see Paul’s com­ment and my re­ply]

In Paul’s clas­sic post What does the uni­ver­sal prior ac­tu­ally look like? he lays out an ar­gu­ment that the uni­ver­sal prior, if it were to be used for im­por­tant de­ci­sions, would likely be ma­lign, giv­ing pre­dic­tions that would effec­tively be un­der the con­trol of alien con­se­quen­tial­ists. He ar­gues for this based on an ‘an­thropic up­date’ the aliens could make that would be difficult to rep­re­sent in a short pro­gram. We can split this up­date into two parts: an ‘im­por­tance up­date’ re­strict­ing at­ten­tion to bits fed into pri­ors used to make im­por­tant de­ci­sions, and what I’m call­ing a ‘lex­i­cal up­date’ which de­pends on the par­tic­u­lar var­i­ant of the uni­ver­sal prior be­ing used. I still be­lieve that the ‘im­por­tance up­date’ would be very pow­er­ful, but I’m not sure any­more about the ‘lex­i­cal up­date’. So in this post I’m go­ing to sum­ma­rize both in my own words then ex­plain my skep­ti­cism to­wards the ‘lex­i­cal up­date’.

As back­ground, note that ‘straight­for­wardly’ spec­i­fy­ing data such as our ex­pe­riences in the uni­ver­sal prior will take far more bits than just de­scribing the laws of physics, as you’ll also need to de­scribe our lo­ca­tion in space­time, an in­put method, and a set of Everett branches(!), all of which to­gether will prob­a­bly take more than 10000 bits(com­pared to the laws alone which likely only take a few hun­dred) Thus, any re­ally short pro­gram(a few hun­dred bits, say) that could some­how pre­dict our ex­pe­riences well would likely have a greater prob­a­bil­ity ac­cord­ing to the uni­ver­sal prior than the ‘straight­for­ward’ ex­pla­na­tion.

Paul’s post ar­gues that there likely do ex­ist such pro­grams. I’m go­ing to fix a refer­ence pre­fix ma­chine U which gen­er­ates a uni­ver­sal prior. The ar­gu­ment goes:

A) there are many long-run­ning pro­grams with short de­scrip­tions ac­cord­ing to U, such as our uni­verse.

B) If other pro­grams are like our uni­verse’s pro­gram, aliens could evolve there and end up tak­ing over their pro­grams.

C) Since their pro­gram has high mea­sure in U, the aliens will plau­si­bly have been se­lected to be mo­ti­vated to con­trol short pro­grams in U.

D) To con­trol U, the aliens could try to ma­nipu­late be­ings us­ing the uni­ver­sal prior who have con­trol over short pro­grams in U (like us, hy­po­thet­i­cally)

E) If the aliens are rea­son­ably mo­ti­vated to ma­nipu­late U, we can sam­ple them do­ing that with few bits.

F) The aliens will now try to out­put sam­ples from Q, the dis­tri­bu­tion over peo­ple us­ing the uni­ver­sal prior to make im­por­tant de­ci­sions(de­ci­sions im­pact­ing short pro­grams in U). They can do this much more effi­ciently than any ‘straight­for­ward’ method. For in­stance, when spec­i­fy­ing which planet we are on, the aliens can re­strict at­ten­tion to planets which even­tu­ally de­velop life, sav­ing a great many bits.

G) The aliens can then choose a low-bit broad­cast chan­nel in their own uni­verse, so the en­tire ma­nipu­la­tive be­hav­ior has a very short de­scrip­tion in U.

H) For a short pro­gram to com­pete with the aliens, it would es­sen­tially need ac­cess to Q. But this seems re­ally hard to spec­ify briefly.

So far I agree. But the post also ar­gues that even a short pro­gram that could sam­ple from Q would still lose badly to the aliens, based on what I’m call­ing a ‘lex­i­cal up­date’, as fol­lows:

I) In prac­tice most peo­ple in U us­ing ‘the uni­ver­sal prior’ won’t use U it­self but one of many var­i­ants U’(differ­ent uni­ver­sal pro­gram­ming lan­guages)

J) Each of those var­i­ants U’ will have their own Q’, the dis­tri­bu­tion over peo­ple mak­ing im­por­tant de­ci­sions with U’. Q is then defined as the av­er­age over all of those var­i­ants(with differ­ent U’ weighted by sim­plic­ity in U)

K) Since short pro­grams in differ­ent U’ look differ­ent from each other, the aliens in those pro­grams will be able to tell which var­i­ant U’ they are likely to be in.

L) The dis­tri­bu­tions Q’ of peo­ple in U us­ing differ­ent var­i­ants U’ all look differ­ent. De­scribing each Q’ given Q should take about as many bits as it takes to spec­ify U’ us­ing U.

M) But the aliens will already know they are in U’, and so can skip that, gain­ing a large ad­van­tage even over Q.

But there’s a prob­lem here. In C) it was ar­gued that aliens in short pro­grams in U will be mo­ti­vated to take over other short pro­grams in U. When we con­di­tion on the aliens ac­tu­ally liv­ing some­where short ac­cord­ing to U’, they are in­stead mo­ti­vated to con­trol short pro­grams in U’. This would re­duce their mo­ti­va­tion to con­trol short pro­grams in U pro­por­tion­ally to the difficulty of de­scribing U in U’, and with less mo­ti­va­tion, it takes more bits to sam­ple their ma­nipu­la­tive be­hav­iors in E). The ad­van­tage they gained in L) over Q was pro­por­tional to the difficulty of de­scribing U’ in U. On av­er­age these effects should can­cel out, and the aliens’ prob­a­bil­ity mass will be com­pa­rable to Q.

The uni­ver­sal prior is still likely ma­lign, as it’s prob­a­bly hard to briefly spec­ify Q, but it no longer seems to me like the aliens would de­ci­sively beat Q. I still feel pretty con­fused about all this so com­ments point­ing out any mis­takes or mis­in­ter­pre­ta­tions would be ap­pre­ci­ated.