Weak arguments against the universal prior being malign

Paul Chris­ti­ano makes the case that if we use the uni­ver­sal prior to make im­por­tant pre­dic­tions, then we will end up as­sign­ing a large amount of prob­a­bil­ity mass to hy­pothe­ses which in­volve in­tel­li­gent agents liv­ing in al­ter­nate uni­verses who have thus-far de­liber­ately made the cor­rect pre­dic­tions so that they might even­tu­ally ma­nipu­late us into do­ing what they want us to do. Paul calls these in­tel­li­gent agents ‘con­se­quen­tial­ists’.

I find ideas like this very difficult to think about clearly, but I have a strong gut-feel­ing that the ar­gu­ment is not cor­rect. I’ve been un­able to form a crisp for­mal ar­gu­ment against Paul’s pro­posal, but be­low I list a few weak rea­sons why the con­se­quen­tial­ists’s prob­a­bil­ity mass in the uni­ver­sal prior might not be as high as Paul sug­gests.

  • Un­nat­u­ral out­put chan­nel: It is prob­a­bly the case that in the vast ma­jor­ity of sim­ple uni­verses which ul­ti­mately spawn in­tel­li­gent life, the most nat­u­ral out­put chan­nel is not ac­cessible to its in­hab­itants. Paul gives an ex­am­ple of such an out­put chan­nel in his post: in a cel­lu­lar au­tomata we could read data by sam­pling the state of the first non-zero cell. The most nat­u­ral thing here would prob­a­bly be to start sam­pling im­me­di­ately from . How­ever, if the au­tomata has sim­ple rules and a sim­ple start­ing state then it will take a very large num­ber of time-steps be­fore con­se­quen­tial­ist life has had time to evolve to the point at which it can start to in­ten­tion­ally ma­nipu­late the out­put cell. As an­other ex­am­ple, take our own uni­verse: if the ‘most nat­u­ral’ out­put chan­nel in our uni­verse cor­re­sponds to a par­tic­u­lar lo­ca­tion then this prob­a­bly isn’t in­side our light-cone right now.

  • Un­nat­u­ral in­put chan­nel: Similar to nat­u­ral out­put chan­nels not nec­es­sar­ily be­ing ac­cessible, of­ten it will also be im­pos­si­ble for a con­se­quen­tial­ist to dis­cern ex­actly what was fed in to her uni­verse’s in­put chan­nel. In the ex­am­ple of a cel­lu­lar au­tomata, the most nat­u­ral in­put chan­nel is prob­a­bly the ini­tial state. This is a prob­lem for the au­tomata’s in­hab­itants be­cause, while know­ing the state of the uni­verse at a par­tic­u­lar time lets you pre­dict the next state, in gen­eral it won’t let you de­duce ex­actly how old the uni­verse is or what its ini­tial con­di­tions were. Another source of difficulty in re­cov­er­ing the data fed into your uni­verse’s in­put chan­nel is that if your uni­verse im­ple­ments some­thing analo­gous to dis­tance/​ve­loc­ity then it in many cases some in­for­ma­tion nec­es­sary to re­cover the data fed into your uni­verse’s in­put chan­nel might be mov­ing away from you too fast for you to ever re­cover it (e.g. a space ship fly­ing away from you at max speed in Con­way’s game of life).

  • Im­plicit com­pu­ta­tional con­straints: A com­plaint many peo­ple have with the uni­ver­sal prior is that it places no con­straints on the amount of com­pute as­so­ci­ated with a par­tic­u­lar hy­poth­e­sis, (mean­ing it al­lows ab­surd hy­poth­e­sis like dae­mons in al­ter­nate uni­verses). It is worth notic­ing that while there is no ex­plicit com­pu­ta­tional penalty, dae­mons in­side the prior are sub­ject to im­plicit com­pu­ta­tional con­straints. If the pro­cess which the al­ter­nate-uni­verse con­se­quen­tial­ists must use to pre­dict the next ob­ser­va­tion we’re about to see re­quires a lot of com­pute, then from the con­se­quen­tial­ist’s per­spec­tive this fact is not ir­rele­vant. This is be­cause (as­sum­ing they care about lots of things, not just con­trol­ling the uni­ver­sal prior) they will per­ceive the cost of the com­pu­ta­tion as a rele­vant ex­pense which must be traded off against their other prefer­ences, even though we don’t per­son­ally care how much com­pute power they use. Th­ese im­plicit com­pu­ta­tional costs can also fur­ther com­pro­mise the con­se­quen­tial­ist’s ac­cess to their uni­verse’s out­put chan­nel. For ex­am­ple con­sider again a sim­ple cel­lu­lar au­tomata such as Con­way’s game of life. Con­way’s game of life is Tur­ing com­plete—it’s pos­si­ble to com­pute an ar­bi­trary se­quence of bits (or simu­late any com­pu­at­able uni­verse) from within the game of life. How­ever, I sus­pect it isn’t pos­si­ble to com­pute an ar­bi­trary se­quence of bits such that this string can be read off by sam­pling a par­tic­u­lar cell once ev­ery time-tick. In a similar vein, while you can in­deed build Minecraft in­side Minecraft, you can’t do it in such a way that the ‘real’ Minecraft world and the ‘simu­lated’ Minecraft world run at the same speed. So con­straints re­lat­ing to speed-of-com­pu­ta­tion fur­ther re­strict the kinds of out­put chan­nels the con­se­quen­tial­ists are able to ma­nipu­late (and if tar­get­ing a par­tic­u­lar out­put chan­nel is very costly then they will have to trade-off be­tween sim­plic­ity of the out­put chan­nel and ex­pense of reach­ing it).

I’m tempted to make fur­ther ar­gu­ments about the un­like­li­ness that any par­tic­u­lar con­se­quen­tial­ist would es­pe­cially care about ma­nipu­lat­ing our Solomonoff in­duc­tor more than any other Solomonoff in­duc­tor in the Teg­mark IV mul­ti­verse, (even af­ter con­di­tion­ing on the im­por­tance of our de­ci­sion and the lan­guage we use to mea­sure com­plex­ity), but I don’t think I fully un­der­stand Paul’s idea of an an­thropic up­date, so there’s a good chance this ob­jec­tion has already been ad­dressed.

All these rea­sons don’t com­pletely elimi­nate dae­mons from the uni­ver­sal prior, but I think they might re­duce their prob­a­bil­ity mass to epistem­i­cally ap­pro­pri­ate lev­els. I’ve re­lied ex­ten­sively on the cel­lu­lar au­tomata case for ex­am­ples and for driv­ing my own in­tu­itions, which might have lead me to over­es­ti­mate the sever­ity of some of the com­plex­ities listed above. Th­ese ideas are su­per weird and I find it very hard to think clearly and pre­cisely about them so I could eas­ily be mis­taken, please point out any er­rors I’m mak­ing.

No nominations.
No reviews.