Anthropics in a Tegmark Multiverse

[paulfchristiano]

I be­lieve that the ex­e­cu­tion of a cer­tain com­pu­ta­tion is a nec­es­sary and suffi­cient con­di­tion for my con­scious ex­pe­rience. Fol­low­ing Teg­mark, by “ex­e­cu­tion” I don’t re­fer to any no­tion of phys­i­cal ex­is­tence—I sus­pect that the math­e­mat­i­cal pos­si­bil­ity of my thoughts im­plies con­scious ex­pe­rience. By ob­serv­ing the world and pos­tu­lat­ing my own rep­re­sen­ta­tive­ness, I con­jec­ture the fol­low­ing mea­sure on differ­ent pos­si­ble ex­pe­riences: the prob­a­bil­ity of any par­tic­u­lar ex­pe­rience drops off ex­po­nen­tially with the com­plex­ity re­quired to spec­ify the cor­re­spond­ing com­pu­ta­tion.

It is typ­i­cal to use some com­plex­ity prior to se­lect a uni­verse, and then to ap­peal to some differ­ent no­tion to han­dle the re­main­ing an­thropic rea­son­ing (to ask: how many be­ings have my ex­pe­riences within this uni­verse?). What I am sug­gest­ing is to in­stead ap­ply a com­plex­ity prior to our ex­pe­riences di­rectly.

If I be­lieve a brain em­body­ing my thoughts ex­ists in some sim­ple uni­verse, then my thoughts can be de­scribed pre­cisely by first de­scribing that uni­verse and then point­ing to the net­work of causal re­la­tion­ships which con­sti­tute my thoughts. If I have seen enough of the uni­verse, then this will be the most con­cise de­scrip­tion con­sis­tent with my ex­pe­riences. If there are many “copies” of that brain within the uni­verse, then it be­comes that much eas­ier to spec­ify my thoughts. In fact, it is easy to check that you re­cover es­sen­tially in­tu­itive an­throp­ics in this way.

This prior has a sig­nifi­cant im­pact on the sta­tus of simu­la­tions. In gen­eral, mak­ing two simu­la­tions of a brain puts twice as much prob­a­bil­ity on the as­so­ci­ated ex­pe­riences. How­ever, we no longer main­tain sub­strate in­de­pen­dence (which I now con­sider a good thing, hav­ing dis­cov­ered that my naive treat­ment of an­throp­ics for simu­la­tions is wildly in­con­sis­tent). The sig­nifi­cance of a par­tic­u­lar simu­la­tion de­pends on how difficult it is to spec­ify (within the sim­ple uni­verse con­tain­ing that simu­la­tion) the causal re­la­tion­ships that rep­re­sent its thoughts. So if we imag­ine the pro­cess of “split­ting” a simu­la­tion run­ning on a com­puter which is two atoms thick, we pre­dict that (at least un­der cer­tain cir­cum­stances) the num­ber of copies dou­bles but the com­plex­ity of spec­i­fy­ing each one in­creases to can­cel the effect.

This prior also gives pre­cise an­swers to an­thropic ques­tions in cos­mol­ogy. Even in an in­finite uni­verse, de­scrip­tion com­plex­ity still an­swers ques­tions such as “how much of you is there? Why aren’t you a Boltz­mann brain?” (of course this still sup­poses that a com­plex­ity prior is ap­pli­ca­ble to the uni­verse).

This prior also, at least in prin­ci­ple, tells you how to han­dle an­throp­ics across quan­tum wor­lds. Either it can ac­count for the Born prob­a­bil­ities (pos­si­bly in con­junc­tion with some ad­di­tional physics, like stray prob­a­bil­ity mass wan­der­ing in from nearby in­co­her­ent wor­lds) or it can’t. In that sense, this the­ory makes a testable “pre­dic­tion.” If it does cor­rectly ex­plain the Born prob­a­bil­ities, then I feel sig­nifi­cantly more con­fi­dence in my un­der­stand­ing of quan­tum me­chan­ics and in this ver­sion of a math­e­mat­i­cal mul­ti­verse. If it doesn’t, then I ten­ta­tively re­ject this ver­sion of a math­e­mat­i­cal mul­ti­verse (ten­ta­tively be­cause there could cer­tainly be more com­pli­cated things still hap­pen­ing in quan­tum me­chan­ics, and I don’t yet know of any satis­fac­tory ex­pla­na­tion for the Born prob­a­bil­ities).

Edit: this idea is ex­actly the same as UDASSA as ini­tially ar­tic­u­lated by Wei Dai. I think it is a shame that the ar­gu­ments aren’t more wide­spread, since it very cleanly re­solves some of my con­fu­sion about simu­la­tions and in­finite cos­molo­gies. My only con­tri­bu­tion ap­pears to be a slightly more con­crete plan for calcu­lat­ing (or failing to calcu­late) the Born prob­a­bil­ities; I will re­port back later about how the com­pu­ta­tion goes.

[jimrandomh]

Un­for­tu­nately, “de­scrip­tion length” does not uniquely iden­tify a value. To get a de­scrip­tion length for a laws-of-physics or a po­si­tion in­side them, you need a de­scrip­tion lan­guage and a met­ric over it. Much has been made of the fact that the com­plex­ity of a laws-of-physics differs by only an ad­di­tive con­stant be­tween de­scrip­tion lan­guages (since you can always pre­fix an in­ter­preter), but un­for­tu­nately that ad­di­tive con­stant is enough to bend the model into con­sis­tency with al­most any ob­served laws-of-physics.

[torekp]

Upvoted—can you please sup­ply links to more de­tailed dis­cus­sion of these points?

[Wei Dai]

Seems like we have a lot of similar ideas/​in­ter­ests. :) This is was my main ap­proach for solv­ing an­thropic rea­son­ing, un­til I gave it up in fa­vor of UDT.

Here are some pre­vi­ous re­lated dis­cus­sions.

• http://​​www.fin­ney.org/​​~hal/​​udassa/​​sum­mary1.html

• http://​​less­wrong.com/​​lw/​​1g4/​​tips_and_tricks_for_an­swer­ing_hard_ques­tions/​​1gwq?c=1

• http://​​less­wrong.com/​​lw/​​py/​​the_born_prob­a­bil­ities/​​141m

[paulfchristiano]

I am more op­ti­mistic than you about how the Born prob­a­bil­ities could arise from in­duc­tion. You seem to sug­gest that any other prob­a­bil­ity rule would be less likely to pro­duce sen­tience, so most ob­server mo­ments will be se­lected ac­cord­ing to the Born rule. This seems ini­tially plau­si­ble, but it is quite un­satis­fy­ing and I am not very con­fi­dent that the ar­gu­ment ac­tu­ally goes through.

I am hop­ing that the num­ber of pos­si­ble speci­fi­ca­tions of any ob­server mo­ment (once the uni­ver­sal wave­func­tion has already been con­cisely de­scribed) de­pends on the squared norm of the pro­jec­tion of the uni­ver­sal wave­func­tion onto the sub­space cor­re­spond­ing to that ob­server mo­ment. For any par­tic­u­lar way of pick­ing out ob­server mo­ments, I think this is es­sen­tially a ge­o­met­ri­cal state­ment which is ei­ther to easy ver­ify or re­fute. In­tu­itively I think it may hold the most nat­u­ral way of spec­i­fy­ing an ob­server mo­ment given a state.

Of course, al­though this would be fairly com­pel­ling, you would still have to rule out other effi­cient ways of spec­i­fy­ing an ob­server mo­ment once you have the uni­ver­sal wave­func­tion.

[Manfred]

Let’s say you flip a coin ten times. Would you ex­pect ten heads to oc­cur more of­ten than 1 in 2^10 in your ex­pe­rience, be­cause it is sim­pler to think “ten heads” than it is to think “HTTHTHHHTH”?

[Plasmon]

Does your point re­main valid if you take a re­al­is­tic dis­tri­bu­tion over coin im­perfec­tions into ac­count?

Pos­si­bly ir­rele­vant calcu­la­tion fol­lows (do we have hide tags? Ap­par­ently not)

Sup­pose we have the sim­plest sort of de­vi­a­tion pos­si­ble: let alpha be a small number

P(10 heads) = (1/​2+alpha)^10

P(HTTHTHHHTH) = (1/​2+alpha)^6*(1/​2-alpha)^4

Re­mark­ably (?)

dP(10 heads) /​dalpha = ⁵⁄₂₅₆ at alpha=0

dP(HTTHTHHHTH) /​dalpha = ¹⁄₂₅₆ at alpha=0

It seems that sim­ple coin de­vi­a­tions (which are by hy­poth­e­sis the most prob­a­ble) have a stronger in­fluence on sim­ple pre­dic­tions such as P(10 heads) than on com­pli­cated pre­dic­tions such as P(HTTHTHHHTH)

[endoself]

Ap­ply­ing this to the real world, the the­ory pre­dicts that I should ex­pect my­self at my cur­rent mo­ment to be Kol­mogorov sim­ple. I don’t feel par­tic­u­larly sim­ple, but this is differ­ent from be­ing sim­ple. There is only strong ev­i­dence against the the­ory if it is prob­a­ble that sim­plic­ity is per­ceived, con­di­tional on it ex­ist­ing.

I think it would be easy for a con­scious be­ing to not per­ceive it sim­plic­ity be­cause my ex­pe­rience with math and sci­ence shows that hu­mans of­ten do not eas­ily no­tice sim­plic­ity be­yond a very low thresh­old. Some be­ings may be be­low this thresh­old, such as the first con­scious be­ing or the most mas­sive con­scious be­ing, but I find it un­likely that be­ings of this type have a prob­a­bil­ity any­where near that of all other con­scious be­ings, es­pe­cially con­sid­er­ing how hard these con­cepts are to make pre­cise.

Us­ing your ex­am­ple of be­ings that ob­serve coin tosses, sim­ple but low-prob­a­bil­ity events may be the eas­iest way to spec­ify some­one, but there could also eas­ily be a way with less com­plex­ity but less ap­par­ent to the ob­server. This seems likely enough that not ob­serv­ing high-prob­a­bil­ity events does not provide ex­cep­tion­ally strong ev­i­dence against the the­ory.

[paulfchristiano]

No.

The sim­plest way to de­scribe ei­ther phe­nomenon—when com­bined with the other ex­pe­rience that leads me to be­lieve there is a uni­verse be­yond my brain—is to de­scribe the uni­verse and point to my brain in­side it. If I saw enough coins come up heads in some con­sis­tent situ­a­tion (for ex­am­ple, when­ever I try and test an­thropic prin­ci­ples) then at some point a lawful uni­verse will cease to be the best ex­pla­na­tion. The ex­act same thing is true for Solomonoff in­duc­tion as well, though the quan­ti­ta­tive de­tails may differ very slightly.

[Manfred]

But or­der­ing over the com­plex­ity of your brain, rather than the uni­verse, is already pos­tu­lat­ing that a lawful uni­verse isn’t the best ex­pla­na­tion. You can’t have your cake and eat it too.

[paulfchristiano]

A lawful uni­verse is the best ex­pla­na­tion for my ex­pe­riences. My ex­pe­rience is em­bod­ied in a par­tic­u­lar cog­ni­tive pro­cess. To de­scribe this pro­cess I say:

“Con­sider the sys­tem satis­fy­ing the law L. To find Paul within that sys­tem, look over here.”

In or­der to de­scribe the ver­sion of me that sees 10 heads in a row, I in­stead have to say:

“Con­sider the sys­tem satis­fy­ing the law L, in which these 10 coins came up heads. To find Paul within that uni­verse, look over here.”

The prob­a­bil­ity of see­ing 10 heads in a row may be slightly higher: adding ad­di­tional ex­pla­na­tions in­creases the prob­a­bil­ity of an ex­pe­rience, and the de­scrip­tion of “ar­bi­trary change” is eas­ier if the change is to make all 10 out­comes H rather than to set the out­comes in some more com­pli­cated way. How­ever, the same effect is pre­sent in Solomonoff in­duc­tion.

There are many more sub­tleties here, and there are uni­verses which in­volve ran­dom­ness in a way where I would pre­dict that HHHHHHHHH is the most likely re­sult from look­ing at 10 coin flips in a row. But the same things hap­pen with Solomonoff in­duc­tion, so they don’t seem worth talk­ing about here.

[Manfred]

Best ex­pla­na­tion by what stan­dard? By the stan­dard where you rank uni­verses from least com­plex to most com­plex! You can­not do two differ­ent rank­ings si­mul­ta­neously.

So then, are you say­ing that you do not think that a sim­plic­ity prior on your brain is a good idea?

[paulfchristiano]

Short­est ex­pla­na­tion for my thoughts. Pre­cisely a sim­plic­ity prior on my brain. There is noth­ing about uni­verse com­plex­ity.

I be­lieve that the short­est ex­pla­na­tion for my thoughts is the one that says “Here is the uni­verse. Within the uni­verse, here is this dude.” This is a valid ex­pla­na­tion for my brain, and it gets longer if I have to mod­ify it to make my brain “sim­pler” in the sense you are us­ing, not shorter.

[Manfred]

No, it doesn’t. Pick­ing be­tween microstates isn’t a “mod­ifi­ca­tion” of the uni­verse, it’s sim­ply talk­ing about the ob­served prob­a­bil­ity of some­thing that already hap­pens all the time.

Although now that I think about it, this ar­gu­ment should ap­ply to more tra­di­tional an­throp­ics as well, if a sim­plic­ity prior is used. And since I’ve done this ex­per­i­ment a few times now, I can say with high con­fi­dence that a strong sim­plic­ity prior is in­cor­rect when flip­ping coins (es­pe­cially when an­throp­i­cally flip­ping coins [which means I did it my­self]), and a max­i­mum en­tropy prior is very close to cor­rect.

[AlephNeil]

I think I can see a flaw.

OK, so your cen­tral idea is to use the com­plex­ity prior for ‘cen­tered wor­lds’ rather than ‘un­cen­tered wor­lds’. A ‘cen­tered world’ here means a “world + poin­ter to the ob­server”.

Now, if I give you a world + a poin­ter to the ob­server then you can tell me ex­actly what the ob­server’s sub­jec­tive state is, right? There­fore, the com­plex­ity of “world + poin­ter to ob­server” is greater than the com­plex­ity of “sub­jec­tive state all by it­self + the iden­tity func­tion”.

There­fore, your ap­proach en­tails that we should give mas­sive weight to the pos­si­bil­ity that solip­sism is cor­rect :-)

ETA: Fixed an im­por­tant er­ror—had writ­ten “less” in­stead of “greater”.

[paulfchristiano]

My ap­proach is defined by solipism. I don’t use the com­plex­ity prior for ‘cen­tered wor­lds’ I just use the com­plex­ity prior for ‘sub­jec­tive state.’

That said, all of the other peo­ple in the world prob­a­bly also have men­tal states which are just as eas­ily de­scribed, so its an empty sort of solipism.

(Also note that my idea is ex­actly the same as Wei Dai’s).

[AlephNeil]

You say:

If I be­lieve a brain em­body­ing my thoughts ex­ists in some sim­ple uni­verse, then my thoughts can be de­scribed pre­cisely by first de­scribing that uni­verse and then point­ing to the net­work of causal re­la­tion­ships which con­sti­tute my thoughts.

There­fore, the com­plex­ity of “uni­verse + poin­ter to the net­work of causal re­la­tion­ships con­sti­tut­ing your thoughts” is greater than or equal to the com­plex­ity of “net­work of causal re­la­tion­ships con­sti­tut­ing your thoughts + the iden­tity func­tion”.

Really, you should just talk about the ‘net­work of causal re­la­tion­ships con­sti­tut­ing your thoughts’ all by it­self. So if Jupiter hasn’t af­fected your thoughts yet, Jupiter doesn’t ex­ist? But what counts as an ‘effect’? And what are the bound­aries of your men­tal state? This gets awfully tricky.

[paulfchristiano]

The is­sue is that you mean a differ­ent thing by “com­plex­ity” then the defi­ni­tion.

How do you de­scribe your thoughts all by them­selves? You could de­scribe the whole phys­i­cal brain and its bound­ary with the world, but that is spec­tac­u­larly com­plex. Sim­pler is to spec­ify the uni­verse (by giv­ing some sim­ple laws which gov­ern it) and then to de­scribe where to find your thoughts in it. This is the short­est com­pu­ta­tional recipe which out­puts a de­scrip­tion of your thoughts.

[AlephNeil]

How do you de­scribe your thoughts all by them­selves?

By de­scribing the ab­stract struc­ture of that ‘net­work of causal re­la­tion­ships’ you were talk­ing about?

Look, there’s a Mas­sive Philo­soph­i­cal Prob­lem here which is “what do you take your thoughts to be?” But what­ever an­swer you give, other than just “a uni­verse plus a poin­ter” I can carry on re­peat­ing my trick.

It sounds as though you want to give the an­swer “an equiv­alence class of uni­verses-plus-poin­t­ers, where (W1, P1) ~ (W2, P2) iff the be­ing at P1 ‘has the same thoughts’ as the be­ing at P2”. But this is no good if we don’t know what “thoughts” are yet.

ETA: Just wanted to say that the post was very in­ter­est­ing, re­gard­less of whether I think I can re­fute it, and I hope LW will con­tinue to see dis­cus­sions like this.

[paulfchristiano]

By de­scribing the ab­stract struc­ture of that ‘net­work of causal re­la­tion­ships’ you were talk­ing about?

So you can de­scribe your brain by say­ing ex­plic­itly what it con­tains, but this is not the short­est pos­si­ble de­scrip­tion in the sense of Kol­mogorov com­plex­ity.

I be­lieve that the short­est way to de­scribe the con­tents of your brain—not your brain sit­ting in­side a uni­verse or any­thing—is to de­scribe the uni­verse (which has lower com­plex­ity than your brain, in the sense that it is the out­put of a shorter pro­gram) and then to point to your brain. This has lower com­plex­ity than try­ing to de­scribe your brain di­rectly.

[AlephNeil]

I un­der­stand what you were try­ing to do a lit­tle bet­ter now.

This has lower com­plex­ity than try­ing to de­scribe your brain di­rectly.

I think that so far you’ve tended to treat this as if it was ob­vi­ous whereas I’ve treated it as if it was ob­vi­ously false, but nei­ther of us has given much in the way of jus­tifi­ca­tion.

Some things to keep in mind:

• Giv­ing a poin­ter to a po­si­tion within some ‘branch’ of the mul­ti­verse could cost a hell of a lot of in­for­ma­tion. (Rather like how spec­i­fy­ing the lo­ca­tion of a book within the Library of Ba­bel offers zero com­pres­sion com­pared to just writ­ing out the book.) I un­der­stand that, if there are lots of copies then, ce­teris paribus, the length of a poin­ter should de­crease at rate log­a­r­ith­mic in the num­ber of copies. But it’s not ob­vi­ous that this re­duces the cost to be­low that of a more ‘di­rect’ de­scrip­tion.

• There are many pos­si­bil­ities other than ‘di­rect, naive, literal, ex­plicit de­scrip­tion’ and de­scrip­tion in terms of ‘our uni­verse + a poin­ter’. For in­stance, one could ap­ply some com­pres­sion al­gorithm to an ex­plicit de­scrip­tion. And here it’s con­ceiv­able that the re­sult could be re­garded as a de­scrip­tion of the men­tal state as sit­ting in­side some larger, sim­pler ‘uni­verse’, but very differ­ent from and much smaller than the ‘real world’. Is it re­ally true that all of gen­eral rel­a­tivity and quan­tum me­chan­ics is im­plicit in the men­tal state of an an­cient thinker like Socrates? I don’t want to say the an­swer is ‘ob­vi­ously not’ - ac­tu­ally I find it ex­tremely difficult to jus­tify ‘yes’ or ‘no’ here. Some of the difficulty is due to the in­de­ter­mi­nacy in the con­cept of “men­tal state”. (And what if we re­place ‘Socrates’ with ‘a cow’?)

• What reper­toire of log­i­cal and/​or phys­i­cal prim­i­tives are per­mit­ted when it comes to writ­ing down a poin­ter? For in­stance, in a uni­verse con­tain­ing just a sin­gle ob­server, can we effi­ciently ‘point to’ the ob­server by de­scribing it as “the unique ob­server”? In our own uni­verse, can we shorten a poin­ter by de­scribing Jones as “the near­est per­son to X” where X is some eas­ily-de­scrib­able land­mark (e.g. a su­per­mas­sive black hole)?

[jhuffman]

I think a poin­ter that effec­tively forces you to com­pute the en­tire pro­gram in or­der to find the ob­ject it refer­ences is still re­duc­ing com­plex­ity based on the defi­ni­tion used. Com­pu­ta­tion­ally ex­pen­sive != com­plex.

[AlephNeil]

Sure, it might be re­duc­ing com­plex­ity, but it might not be. Con­sider the Library of Ba­bel ex­am­ple, and bear in mind that a brain-state has a ton of ex­tra in­for­ma­tion over and above the ‘men­tal state’ it sup­ports. (Though strictly speak­ing this de­pends on the no­tion of ‘men­tal state’, which is in­de­ter­mi­nate.)

Also, we have to ask “re­duc­ing com­plex­ity rel­a­tive to what?” (As I said above, there are many pos­si­bil­ities other than “literal de­scrip­tion” and “our uni­verse + poin­ter”.)

[cousin_it]

This is a new idea to me. I like it a lot.

One way to make it even­tu­ally ex­plain the Born prob­a­bil­ities is to also throw away con­ti­nu­ity of ex­pe­rience and in­stead use some­thing like ASSA: “your cur­rent ob­server-mo­ment is a ran­dom sam­ple from all ob­server-mo­ments”, where the “mea­sure” of each ob­server-mo­ment is de­ter­mined by its de­scrip­tion com­plex­ity as you sug­gest. Then you could look at the uni­ver­sal wave­func­tion (with­out pos­tu­lat­ing the Born prob­a­bil­ities) and note that point­ing out a hu­man ob­server-mo­ment af­ter a split has oc­curred re­quires 1 bit more in­for­ma­tion than point­ing out the ver­sion be­fore the split. I have no idea if it’s true, but it sounds quite plau­si­ble to me.

[Vladimir_Nesov]

What are you us­ing these “prob­a­bil­ities” for? Prob­a­bil­ities of what? What is the in­tended mean­ing of a prob­a­bil­ity-defi­ni­tion you are try­ing to make here? Can you taboo “prob­a­bil­ity” and de­scribe mo­ti­va­tion be­hind the idea you’re de­scribing (as op­posed to other con­no­ta­tions evoked by the word)?

[paulfchristiano]

Couldn’t you ask the same ques­tion of any UDT agent? It seems like messy philo­soph­i­cal ter­ri­tory, but that peo­ple here im­plic­itly wade into fre­quently.

Here is what I think I mean: I can con­trol log­i­cal facts, in­so­far as they de­pend on my thoughts, but the ques­tion is: what do I want to achieve by that con­trol? I care about the con­tent of con­scious ex­pe­rience, but what ex­actly is it that I value?

I have a prefer­ence over the pos­si­ble ob­server-mo­ments which I per­son­ally may ex­pe­rience—I can con­sider two prob­a­bil­ity dis­tri­bu­tions over ob­server-mo­ments and de­cide which one it would rather ex­pe­rience. A prob­a­bil­ity dis­tri­bu­tion is a men­tal con­struct which I need to give as an in­put to my prefer­ence or­der­ing.

The ques­tion is: how do I use this built in no­tion of prefer­ence to dis­crim­i­nate be­tween two re­s­olu­tions of my un­cer­tainty about log­i­cal facts? My plan is to define a prob­a­bil­ity dis­tri­bu­tion over ob­server-mo­ments and sub­ject this dis­tri­bu­tion to my in­tu­itive prefer­ences. Then I can choose to de­ter­mine log­i­cal facts to make this sin­gle fixed prob­a­bil­ity dis­tri­bu­tion over ob­server-mo­ments as de­sir­able as pos­si­ble.

[Wei Dai]

I have a prefer­ence over the pos­si­ble ob­server-mo­ments which I per­son­ally may ex­pe­rience—I can con­sider two prob­a­bil­ity dis­tri­bu­tions over ob­server-mo­ments and de­cide which one it would rather ex­pe­rience.

I’m cu­ri­ous what your thoughts are on this post.

Also, here are a cou­ple of other prob­lems that I ran into, which I think you might be in­ter­ested in, or have some ideas about.

• If you use a UTM-based dis­tri­bu­tion, which UTM do you choose? Is there a no­tion of com­plex­ity that is not rel­a­tive to some­thing more or less ar­bi­trary?

• Us­ing a UTM-based dis­tri­bu­tion seems to im­ply ig­nor­ing all the copies of you who are liv­ing in un­com­putable wor­lds. It would be nice to have a com­plex­ity-based mea­sure over all of math, but that seems im­pos­si­ble.

[paulfchristiano]

I was very con­fused about iden­ti­cal in­de­pen­dent copies be­fore. Right now the view given here is the best one I have thought of—more in­de­pen­dent copies are more sig­nifi­cant, just like copies run­ning on more eas­ily speci­fied sub­strates. In this view copy im­mor­tal­ity has no value—there is no differ­ence be­tween 2 copies with prob­a­bil­ity ¹⁄₂ and 1 copy with prob­a­bil­ity 1.

I have no idea how to choose a no­tion of com­plex­ity, ei­ther amongst UTMs or over some broader class of de­scrip­tions. I hope that at some point I will en­counter a good ar­gu­ment for one choice or an­other, but I don’t yet know of any and its not clear why there would be a good ar­gu­ment.

[Vladimir_Nesov]

Couldn’t you ask the same ques­tion of any UDT agent?

UDT has an­swers. “Prob­a­bil­ity” there plays a par­tic­u­lar role in the de­ci­sion al­gorithm, and you could name that el­e­ment with a differ­ent word while keep­ing the role. They are prob­a­bil­ities of given pro­gram pro­duc­ing given ex­e­cu­tion his­tory, in­ferred given the as­sump­tion that the agent (as a pro­gram) performs given ac­tion. An an­swer of this form would clar­ify your us­age.

I can con­trol log­i­cal facts, in­so­far as they de­pend on my thoughts

Also phys­i­cal facts, which is the way you ac­tu­ally think about phys­i­cal facts, us­ing the ob­ser­va­tions of oth­er­wise opaque log­i­cal speci­fi­ca­tion.

I have a prefer­ence over the pos­si­ble ob­server-mo­ments which I per­son­ally may experience

This is un­clear, since you also have prefer­ence about what hap­pens with the world, and then why con­sider some ar­bi­trary bound­ary that is “you” spe­cially?

I can con­sider two prob­a­bil­ity dis­tri­bu­tions over ob­server-mo­ments and de­cide which one it would rather ex­pe­rience. A prob­a­bil­ity dis­tri­bu­tion is a men­tal con­struct which I need to give as an in­put to my prefer­ence or­der­ing.

Very un­clear. Why would you priv­ilege “ex­pe­rienc­ing” some­thing as a crite­rion for de­ci­sion-mak­ing? Why do you ex­pect a prefer­ence that needs an ar­gu­ment of this par­tic­u­lar form?

For ex­am­ple, how do you ex­press prefer­ence about other hu­man-like agents in the wor­lds you can in­fluence? What about situ­a­tions like ab­sent-minded driver where many usual uses of prob­a­bil­ities break down?

[paulfchristiano]

This is un­clear, since you also have prefer­ence about what hap­pens with the world, and then why con­sider some ar­bi­trary bound­ary that is “you” spe­cially?

I be­lieve my only prefer­ences are over ex­pe­riences. I care noth­ing about a world with­out ob­servers, and I care about our world ex­actly in­so­far as the world has an effect on the ex­pe­riences of cer­tain ob­servers.

I am biolog­i­cally given an in­tu­itive prefer­ence func­tion over prob­a­bil­ity dis­tri­bu­tions of mo­ments that I would per­son­ally ex­pe­rience. The ba­sis of my moral­ity is to use this prefer­ence to try and make in­ter­per­sonal util­ity com­par­i­sons, al­though of course there are many many prob­lems. Put­ting prob­a­bil­ity dis­tri­bu­tions over ob­server mo­ments is a crit­i­cal first step in this pro­gram.

[AlephNeil]

Put­ting prob­a­bil­ity dis­tri­bu­tions over ob­server mo­ments is a crit­i­cal first step in this pro­gram.

There are some prob­lems with the no­tion of ‘ob­server mo­ments’. I’m in­clined to think they are un­re­solv­able, but per­haps you have some ideas for how to tackle them.

I’ve already men­tioned the prob­lem of the ‘bound­ary’ of a sub­jec­tive state. For in­stance, con­sider an old mem­ory which it would take you a long time to ‘dredge up’. How is that differ­ent in prin­ci­ple from hav­ing some in­for­ma­tion writ­ten down some­where in a note­book in front of you? Is that old mem­ory part of your ‘ob­server mo­ment’? (But then how about a slightly fresher mem­ory)? Is the note­book? (But then how about that thick refer­ence book on your shelf)? It seems ob­vi­ous to me that there’s no prin­ci­pled line to be drawn here.

Then there’s the prob­lem of whether a given sys­tem is an ob­server or not. For in­stance, Den­nett is no­to­ri­ous for (cor­rectly!) at­tribut­ing ‘in­ten­tional states’ to a ther­mo­stat: you can view it as an agent who be­lieves the tem­per­a­ture is x and wants it to be y. But is a ther­mo­stat an ob­server? Pre­sum­ably not, but again it seems that there’s no prin­ci­pled line to be drawn be­tween ther­mostats and peo­ple.

And then there’s the prob­lem of ‘how many ob­servers’. E.g. Is a split-brain pa­tient two ob­servers or one? How about an or­di­nary per­son? How much of the cor­pus cal­lo­sum needs to be sev­ered to get two ob­servers?

Fi­nally, if A is much much clev­erer, more alert, and knowl­edge­able than B then A ought to have a greater den­sity of ‘ob­server mo­ments’ than B? But ex­actly how much greater? The idea that there’s a prin­ci­pled way of de­ter­min­ing this seems overop­ti­mistic.

(Go­ing a lit­tle off topic: I’ve ac­tu­ally been think­ing along vaguely similar lines to you just lately. I’ve been try­ing to de­vise an ap­proach to the mind-body prob­lem in terms of “per­spec­tives”. My over­all goal was to try to ‘do jus­tice to’ the com­mon in­tu­ition that a sub­jec­tive point of view must be de­ter­mi­nate and can­not only half-ex­ist, while main­tain­ing the sub­stance of Den­nett’s po­si­tion which im­plies that there may be no fact of the mat­ter as to whether a per­son is con­scious and what they’re con­scious of. My key idea is that (a) there ex­ist gen­uine facts of the form “From per­spec­tive P, such-and-such would be con­sciously ex­pe­rienced” but (b) the per­spec­tives them­selves do not ex­ist—they’re not part of the “state of the world”. (In­stead, they’re “per­spec­tives from which the state of the world man­i­fests it­self”). An anal­ogy would be how, in math­e­mat­i­cal logic, we have this du­al­ity be­tween the­o­ries and mod­els, or more broadly, be­tween syn­tax and se­man­tics. The “syn­tax side” en­ables you to state and prove things about stuff that ex­ists, but the syn­tax it­self doesn’t ex­ist. (In the same way that there’s no such set as ZFC.)

My ‘per­spec­tives’ are es­sen­tially the same as your ‘poin­t­ers-to-ob­servers’. How­ever, I’d want to stress the fact that per­spec­tives are ubiquitous—e.g. you can take the per­spec­tive of a rock, or a ther­mo­stat. And you can take the per­spec­tive of a per­son in many differ­ent ways, with no fact of the mat­ter about which is right, or even whether it’s right to take one. (But from any given per­spec­tive, all the sub­jec­tive facts are nice and ‘de­ter­mi­nate’.)

It never oc­curred to me to try to con­sider the Kol­mogorov com­plex­ities of per­spec­tives. It’s an in­ter­est­ing idea, but it’s hard to wrap one’s head around given that there are an un­limited num­ber of ways of defin­ing the same per­son’s per­spec­tive.)

[Vladimir_Nesov]

I be­lieve my only prefer­ences are over ex­pe­riences. I care noth­ing about a world with­out ob­servers, and I care about our world ex­actly in­so­far as the world has an effect on the ex­pe­riences of cer­tain ob­servers.

How do you know this? (What do you think you know, how do you think you know it?) Your brain might con­tain this be­lief. Should you draw a con­clu­sion from this fact that what the be­lief claims is true? Pre­sum­ably, if your brain con­tained a be­lief that 15+23=36, you’ll have a pro­cess that would not stop at ac­cept­ing this claim just be­cause your brain claimed it’s so, you’ll be able to do bet­ter. There is no mag­i­cal truth-ma­chine in your mind, ev­ery­thing should be sus­pect, un­der­stood and be­lieved stronger only when fur­ther re­flec­tion per­mits.

What “cer­tain ob­servers”? You were talk­ing only about your own ex­pe­riences. How do you take into ac­count ex­pe­riences of other agents? (Or even your­self at other mo­ments, or given al­ter­na­tive ob­ser­va­tions, im­perfect copies.)

[paulfchristiano]

How do I know what I care about? I don’t know.

How would I learn that I care about things other than ob­servers? Where could I find such a fact other than in my mind? I think this has ba­si­cally wan­dered into a metaethics dis­cus­sion. It is a fact that I feel noth­ing about wor­lds with­out ob­servers. Per­haps if you gave me an ar­gu­ment to care I would.

By “cer­tain ob­servers” I only meant to re­fer to “ob­servers whose ex­pe­riences are af­fected by the world”.

[Vladimir_Nesov]

How do I know what I care about? I don’t know.

Then you shouldn’t be cer­tain. Beliefs of un­known ori­gin tend to be un­re­li­able. They also don’t al­low figur­ing out what ex­actly they mean.

How would I learn that I care about things other than ob­servers? Where could I find such a fact other than in my mind?

Con­sider the above ex­am­ple of be­liev­ing that 15+23=36. You can find the facts al­low­ing you to cor­rect the er­ror el­se­where in your mind, they are just be­liefs other than the one that claims 15+23 to be 36. You can also con­sult a calcu­la­tor, some­thing not nor­mally con­sid­ered part of your mind. You can even ask me.

By “cer­tain ob­servers” I only meant to re­fer to “ob­servers whose ex­pe­riences are af­fected by the world”.

This doesn’t help. I don’t see how your al­gorithm dis­cussed in the post would rep­re­sent car­ing about any­thing other than ex­actly its own cur­rent state of mind, which was my ques­tion.

[AlephNeil]

This is awe­some—the ex­change re­minds me of this di­alogue by Smul­lyan. :-)

[paulfchristiano]

Then you shouldn’t be cer­tain.

I’m cer­tainly not. Like I said, if you have any ar­gu­ments I ex­pect they could change my opinion.

I don’t see how your al­gorithm dis­cussed in the post would rep­re­sent car­ing about any­thing other than ex­actly its own cur­rent state of mind, which was my ques­tion.

There are other ob­servers with low com­plex­ities (for ex­am­ple, other hu­mans). I can imag­ine the pos­si­bil­ity of be­ing trans­formed into one of them with prob­a­bil­ity de­pend­ing on their com­plex­ity, and I can use my in­tu­itive prefer­ences to make de­ci­sions which make that imag­ined situ­a­tion as good as pos­si­ble. In what other sense would I care about any­thing?

[Vladimir_Nesov]

Then you shouldn’t be cer­tain.

I’m cer­tainly not. Like I said, if you have any ar­gu­ments I ex­pect they could change my opinion.

No, you are talk­ing about a differ­ent prop­erty of be­liefs, lack of sta­bil­ity to new in­for­ma­tion. I claim that be­cause of lack of re­flec­tive un­der­stand­ing of the ori­gins of the be­lief, you cur­rently shouldn’t be cer­tain, with­out any ad­di­tional ob­ject-level ar­gu­ments point­ing out spe­cific prob­lems or ar­gu­ments for an in­com­pat­i­ble po­si­tion.

There are other ob­servers with low com­plex­ities (for ex­am­ple, other hu­mans). I can imag­ine the pos­si­bil­ity of be­ing trans­formed into one of them with prob­a­bil­ity de­pend­ing on their com­plex­ity, and I can use my in­tu­itive prefer­ences to make de­ci­sions which make that imag­ined situ­a­tion as good as pos­si­ble.

I see. I think this whole line of in­ves­ti­ga­tion is very con­fused.

[paulfchristiano]

No, you are talk­ing about a differ­ent prop­erty of be­liefs, lack of sta­bil­ity to new in­for­ma­tion. I claim that be­cause of lack of re­flec­tive un­der­stand­ing of the ori­gins of the be­lief, you cur­rently shouldn’t be cer­tain, with­out any ad­di­tional ob­ject-level ar­gu­ments point­ing out spe­cific prob­lems or ar­gu­ments for an in­com­pat­i­ble po­si­tion.

I don’t quite un­der­stand. I am not cur­rently cer­tain, in the way I use the term. The way I think about moral ques­tion is by imag­ing some ex­trap­o­lated ver­sion of my­self, who has thought for long enough to ar­rive at sta­ble be­liefs. My con­fi­dence in a moral as­ser­tion is syn­ony­mous with my con­fi­dence that it is also held by this ex­trap­o­lated ver­sion of my­self. Then I am cer­tain of a view pre­cisely when my view is sta­ble.

In what other way can I be cer­tain or un­cer­tain?

[Vladimir_Nesov]

You can come to differ­ent con­clu­sions de­pend­ing on fu­ture ob­ser­va­tions, for ex­am­ple, in which case fur­ther re­flec­tion would not move your level of cer­tainty, the be­lief would be sta­ble, and yet you’d re­main un­cer­tain. For ex­am­ple, con­sider your be­lief about the out­come of a fu­ture coin toss: this be­lief is sta­ble un­der re­flec­tion, but doesn’t claim cer­tainty.

Gen­er­ally, there are many ways in which you can (or should) make de­ci­sions or come to con­clu­sions, your whole de­ci­sion prob­lem, all heuris­tics that make up your mind, can have a hand in de­cid­ing how any given de­tail of your mind should be.

(Also, be­ing cer­tain for the rea­son that you don’t ex­pect to change your mind sounds like a bad idea, this could li­cense ar­bi­trary be­liefs, since the fu­ture pro­cess of po­ten­tially chang­ing your mind that you’re think­ing about could be mak­ing the same calcu­la­tion, lock­ing in into a be­lief with no jus­tifi­ca­tion other than it­self. This doesn’t ob­vi­ously work this way only be­cause you re­tain other, healthy rea­sons for mak­ing con­clu­sions, so this par­tic­u­lar wrong rit­ual washes out.)

[Vladimir_Nesov]

What’s “uni­verse” and what’s “ex­pe­rience”? What are you us­ing prob­a­bil­ity of what for?

Sup­pos­ing you as­sume some sense of ex­pected util­ity max­i­miza­tion, what kinds of things have util­ity and prob­a­bil­ity? Also, for de­ci­sion-mak­ing you need mod­els of con­se­quences of pos­si­ble de­ci­sions, all of which ex­cept the ac­tual one are not ac­tu­ally made, in par­tic­u­lar mod­els of prob­a­bil­ity have to be able to de­pend on your pos­si­ble de­ci­sions. Fo­cus­ing on “ex­pe­rience” doesn’t ob­vi­ously give a way of ad­dress­ing that.

[endoself]

This idea has been dis­cussed be­fore and I have not heard any par­tic­u­larly com­pel­ling ev­i­dence for or against it (see my re­ply to Man­fred). I be­lieve it was origi­nally due to Wei Dai.

[paulfchristiano]

I don’t see how this can be ag­nos­tic on the Born prob­a­bil­ities. Did this ques­tion come up be­fore? Was the con­clu­sion sim­ply “its too hard to tell”?

[endoself]

I don’t re­call the sug­ges­tion of us­ing the Born prob­a­bil­ities as a test ever hav­ing been dis­cussed.

I agree that this could po­ten­tially sup­ply strong ev­i­dence one way or an­other, but I do not see any way of de­riv­ing the Born prob­a­bil­ities and it seems un­likely that it would be easy to prove that de­duc­ing the Born prob­a­bil­ities is im­pos­si­ble, be­cause the deriva­tion might de­pend on some un­known as­pect of physics or con­scious­ness.

[endoself]

I have heard Wei Dai dis­cuss this idea (I be­lieve he came up with it).