# What Are Probabilities, Anyway?

In Prob­a­bil­ity Space & Au­mann Agree­ment, I wrote that prob­a­bil­ities can be thought of as weights that we as­sign to pos­si­ble world-his­to­ries. But what are these weights sup­posed to mean? Here I’ll give a few in­ter­pre­ta­tions that I’ve con­sid­ered and held at one point or an­other, and their prob­lems. (Note that in the pre­vi­ous post, I im­plic­itly used the first in­ter­pre­ta­tion in the fol­low­ing list, since that seems to be the main­stream view.)

1. Only one pos­si­ble world is real, and prob­a­bil­ities rep­re­sent be­liefs about which one is real.

• Which world gets to be real seems ar­bi­trary.

• Most pos­si­ble wor­lds are life­less, so we’d have to be re­ally lucky to be al­ive.

• We have no in­for­ma­tion about the pro­cess that de­ter­mines which world gets to be real, so how can we de­cide what the prob­a­bil­ity mass func­tion p should be?

2. All pos­si­ble wor­lds are real, and prob­a­bil­ities rep­re­sent be­liefs about which one I’m in.

• Be­fore I’ve ob­served any­thing, there seems to be no rea­son to be­lieve that I’m more likely to be in one world than an­other, but we can’t let all their weights be equal.

3. Not all pos­si­ble wor­lds are equally real, and prob­a­bil­ities rep­re­sent “how real” each world is. (This is also some­times called the “mea­sure” or “re­al­ity fluid” view.)

• Which wor­lds get to be “more real” seems ar­bi­trary.

• Be­fore we ob­serve any­thing, we don’t have any in­for­ma­tion about the pro­cess that de­ter­mines the amount of “re­al­ity fluid” in each world, so how can we de­cide what the prob­a­bil­ity mass func­tion p should be?

4. All pos­si­ble wor­lds are real, and prob­a­bil­ities rep­re­sent how much I care about each world. (To make sense of this, re­call that these prob­a­bil­ities are ul­ti­mately mul­ti­plied with util­ities to form ex­pected util­ities in stan­dard de­ci­sion the­o­ries.)

• Which wor­lds I care more or less about seems ar­bi­trary. But per­haps this is less of a prob­lem be­cause I’m “al­lowed” to have ar­bi­trary val­ues.

• Or, from an­other per­spec­tive, this drops an­other an­other hard prob­lem on top of the pile of prob­lems called “val­ues”, where it may never be solved.

As you can see, I think the main prob­lem with all of these in­ter­pre­ta­tions is ar­bi­trari­ness. The un­con­di­tioned prob­a­bil­ity mass func­tion is sup­posed to rep­re­sent my be­liefs be­fore I have ob­served any­thing in the world, so it must rep­re­sent a state of to­tal ig­no­rance. But there seems to be no way to spec­ify such a func­tion with­out in­tro­duc­ing some in­for­ma­tion, which any­one could in­fer by look­ing at the func­tion.

For ex­am­ple, sup­pose we use a uni­ver­sal dis­tri­bu­tion, where we be­lieve that the world-his­tory is the out­put of a uni­ver­sal Tur­ing ma­chine given a uniformly ran­dom in­put tape. But then the dis­tri­bu­tion con­tains the in­for­ma­tion of which UTM we used. Where did that in­for­ma­tion come from?

One could ar­gue that we do have some in­for­ma­tion even be­fore we ob­serve any­thing, be­cause we’re prod­ucts of evolu­tion, which would have built some use­ful in­for­ma­tion into our genes. But to the ex­tent that we can trust the prior speci­fied by our genes, it must be that evolu­tion ap­prox­i­mates a Bayesian up­dat­ing pro­cess, and our prior dis­tri­bu­tion ap­prox­i­mates the pos­te­rior dis­tri­bu­tion of such a pro­cess. The “prior of evolu­tion” still has to rep­re­sent a state of to­tal ig­no­rance.

Th­ese con­sid­er­a­tions lead me to lean to­ward the last in­ter­pre­ta­tion, which is the most tol­er­ant of ar­bi­trari­ness. This in­ter­pre­ta­tion also fits well with the idea that ex­pected util­ity max­i­miza­tion with Bayesian up­dat­ing is just an ap­prox­i­ma­tion of UDT that works in most situ­a­tions. I and oth­ers have already mo­ti­vated UDT by con­sid­er­ing situ­a­tions where Bayesian up­dat­ing doesn’t work, but it seems to me that even if we set those aside, there is still rea­son to con­sider a UDT-like in­ter­pre­ta­tion of prob­a­bil­ity where the weights on pos­si­ble wor­lds rep­re­sent how much we care about those wor­lds.

• In or­der an­swer ques­tions like “What are X, any­way?”, we can (phe­nomenolog­i­cally) turn the ques­tion into some­thing like “What can we do with X?” or “What con­se­quences does X have?”

For ex­am­ple, con­sider the ques­tion “What are or­dered pairs, any­way?”. Some­times you see “defi­ni­tions” of or­dered pairs in terms of set the­ory. Wikipe­dia says that the stan­dard defi­ni­tion of or­dered pairs is:

(a, b) := {{a}, {a, b}}

Many math­e­mat­i­ci­ans find this “defi­ni­tion” un­satis­fac­tory, and view it not as a defi­ni­tion, but an en­cod­ing or trans­la­tion. The cat­e­gory-the­o­retic no­tion of a product might be more satis­fac­tory. It pins down the prop­er­ties that the or­dered pair already had be­fore the “defi­ni­tion” was pro­posed and in what sense ANY con­struc­tion with those prop­er­ties could be used. Lambda calcu­lus has a cou­ple con­struc­tions that look su­perfi­cially quite differ­ent from the set-the­ory ones, but satisfy the cat­e­gory-the­o­retic re­quire­ments.

I guess this is a re­sponse at the meta level, recom­mend­ing this sort of “phe­nomenolog­i­cal” lens as the way to re­solve these sort of ques­tions.

• Lambda calcu­lus has a cou­ple con­struc­tions that look su­perfi­cially quite differ­ent from the set-the­ory ones, but satisfy the cat­e­gory-the­o­retic re­quire­ments.

… as does the set-the­o­retic one.

ETA: Now that I read more closely, you didn’t im­ply oth­er­wise.

• This word “pos­si­ble” car­ries a LOT of hid­den bag­gage. If math tells us any­thing its that LOTS of things SEEM pos­si­ble to us be­cause we aren’t log­i­cally om­ni­scient but aren’t re­ally pos­si­ble.

While we’re at it, how about we drop “wor­lds” from the mix. I don’t think it adds any­thing. If we re­place it with “in­for­ma­tion flows” do things work bet­ter?

• Do you mean some­thing pre­cise by “in­for­ma­tion flows”?

• Pos­si­ble world is a stan­dard term in sev­eral re­lated fields, such as philos­o­phy and lin­guis­tics. Are you ar­gu­ing against my par­tic­u­lar us­age, or all us­age of the term in gen­eral?

• “Wor­lds” ap­par­ently means pretty-much what it means in the MWI.

• Your get­ting your­self in trou­ble be­cause you as­sume that puz­zling ques­tions must have deep an­swers when usu­ally the ques­tion it­self is flawed or mis­lead­ing. In this case there just seems to be a need for any ex­pla­na­tion of the kind you offer nor would be of any use any­way.

Th­ese ‘ex­pla­na­tions’ you offer of prob­a­bil­ity aren’t re­ally ex­plain­ing any­thing. Cer­tainly we do suc­ces­fully use prob­a­bil­ity to rea­son about sys­tems that be­have in a de­ter­minis­tic clas­si­cal fash­ion (rol­ling dice prob­a­bly counts). No mat­ter what sort of prob­a­bil­ity you be­lieve in you have to ex­plain that ap­pli­ca­tion. So in­tro­duc­ing ‘ob­jec­tive’ prob­a­bil­ity merely adds things we need to ex­plain (pos­si­ble wor­lds etc..).

The cor­rect ap­proach is to step back and ask what is it that needs ex­plain­ing. Well prob­a­bil­ity is re­ally noth­ing but a fancy way of count­ing up out­comes. So once we jus­tify de­scribing the world in a prob­a­bil­is­tic fash­ion (even when it’s de­ter­minis­tic in some sense) the ap­pli­ca­tion of math­e­mat­i­cal in­fer­ence to re­for­mu­late that de­scrip­tion in more use­ful ways is un­trou­bling. In other words if it’s rea­son­able to model rol­ling two six sided dice as be­ing in­de­pen­dent uniformly ran­dom vari­ables on 1...6 count­ing up the com­bi­na­tions and say­ing there is a 16 chance of get­ting a 7 doesn’t raise any new difficul­ties.

So the ques­tion just comes down to is it rea­son­able of us to model the world us­ing ran­dom vari­ables?. I mean one might worry that some wor­lds were deeply ‘tricky’ in that al­most always when it ap­peared two ob­jects be­haved like in­de­pen­dent ran­dom vari­ables in re­al­ity there was some hid­den cor­re­la­tion that would even­tu­ally pop out to bite you in the ass and then once you’d taken that cor­re­la­tion into ac­count an­other one would bite you and so on and so on.

But if you think about it for awhile this isn’t re­ally so much a ques­tion about the na­ture of the world as it is a purely math­e­mat­i­cal ques­tion. If we keep fac­tor­ing out by our best pre­dic­tions will the re­main­ing un­ac­counted for vari­a­tion in out­comes ap­pear to be ran­dom, i.e., make mod­el­ing it as ran­dom vari­ables an ac­cu­rate way to make pre­dic­tions? Well that’s ac­tu­ally kinda com­pli­cated, I have a the­o­rem (well tiny tweak of some­one else’s the­o­rem plus in­ter­prata­tion) which I be­lieve says that yes in­deed it must work this way. I won’t go into it here but let me just say one thing to con­vince you of it’s plau­si­bil­ity.

Ba­si­cally the ar­gu­ment is that things only fail to look ran­dom be­cause we no­tice a more ac­cu­rate way of pre­dict­ing their be­hav­ior. The only ev­i­dence for a se­quence of ob­ser­va­tions failing to be ran­dom ac­cord­ing to the sup­posed dis­tri­bu­tion would be a pat­tern in the ob­ser­va­tions not cap­tured by R so would in turn yield a more ac­cu­rate dis­tri­bu­tion. So ba­si­cally the claim is that we can always sim­ply di­vide up any ob­serv­able into the part we can pre­dict (i.e. a dis­tri­bu­tion of out­comes) and the part we can’t. Once you mod out by the part you can pre­dict by defin­tion any­thing left is to­tally un­pre­dictable to you (e.g. com­putable ma­chines) and thus can’t de­tectably fail to look ran­dom ac­cord­ing to it’s dis­tri­bu­tion since that would be a bet­ter pre­dic­tion.

This isn’t rigor­ous (it’s com­pli­catd) but the point is that Ran­dom­ness is noth­ing but our in­abil­ity to make any bet­ter predictions

• All pos­si­ble wor­lds are real, and prob­a­bil­ities rep­re­sent how much I care about each world.

Right, so maybe we need to re­think this whole ra­tio­nal­ity thing, then? I mean, since there are pos­si­ble wor­lds where god ex­ists, un­der this view, the only differ­ence be­tween a cre­ation­ist and a ra­tio­nal athe­ist is one of taste?

To me, the god world seems much eas­ier to deal with and more pleas­ant. So why not shun ra­tio­nal­ity all to­gether if prob­a­bil­ities are ac­tu­ally ar­bi­trary—if think­ing it re­ally does make it so?

• In this view, ra­tio­nal­ity doesn’t play a role in choos­ing the ini­tial weights on the pos­si­ble uni­verses. That job would be handed over to moral philos­o­phy, just like choos­ing the right util­ity func­tion already is.

So why not shun ra­tio­nal­ity all to­gether if prob­a­bil­ities are ac­tu­ally ar­bi­trary—if think­ing it re­ally does make it so?

No, think­ing it doesn’t make it so. Even in this view, the right be­liefs and de­ci­sions aren’t ar­bi­trary, be­cause they de­pend in a lawful way on your prefer­ences. You still want to be ra­tio­nal in or­der to make the best de­ci­sions to satisfy your prefer­ences.

• Even in this view, the right be­liefs and de­ci­sions aren’t ar­bi­trary, be­cause they de­pend in a lawful way on your prefer­ences.

Right, but I don’t ac­tu­ally have a strong prefer­ence for the sim­plic­ity prior that sci­ence uses: if I can just choose what kind of re­al­ity to en­dorse—and there is re­ally no fact of the mat­ter about which one is real—it seems silly to en­dorse the re­al­ity based on the oc­cam prior of sci­ence. Ac­cord­ing to sci­ence—i.e. ac­cord­ing to the prob­a­bil­ity dis­tri­bu­tion you get from up­dat­ing the com­plex­ity/​oc­cam prior with the ev­i­dence—the world is al­lowed to do lots of hor­rible things to me, like kill me.

It would be much more pleas­ant to en­dorse some other prior—for ex­am­ple, one where ev­ery­thing just hap­pens to work out to match my prefer­ences—the “wish­ful think­ing” prior.

In gen­eral, if there is no fact of the mat­ter about what is real, then why would any­one bother to en­dorse any­thing other than their own per­sonal wish­ful think­ing as real? It would seem to be ir­ra­tional not to.

• It would be much more pleas­ant to en­dorse some other prior—for ex­am­ple, one where ev­ery­thing just hap­pens to work out to match my prefer­ences—the “wish­ful think­ing” prior.

Pre­sum­ably you don’t do that be­cause that’s not your ac­tual prior—you don’t just care about one par­tic­u­lar pos­si­ble world where things hap­pen to turn out ex­actly the way you want. You also care about other pos­si­ble wor­lds and want to make de­ci­sions in ways that make those wor­lds bet­ter.

In gen­eral, if there is no fact of the mat­ter about what is real, then why would any­one bother to en­dorse any­thing other than their own per­sonal wish­ful think­ing as real?

It would be for the same rea­son that you don’t change your util­ity func­tion to give ev­ery­thing an in­finite util­ity.

• you don’t just care about one par­tic­u­lar pos­si­ble world where things hap­pen to turn out ex­actly the way you want.

Pre­sum­ably there are in­finitely many pos­si­ble wor­lds where things hap­pen to turn out ex­actly the way I want: I care about some small finite sub­set of the world, and the rest is al­lowed to vary. Why should I ex­pend en­ergy wor­ry­ing about one par­tic­u­lar in­finity of wor­lds that are hard to op­ti­mize when I have already got in­finitely many where I win eas­ily or by de­fault?

There are pre­sum­ably also in­finitely many pos­si­ble wor­lds where all va­ri­eties of bizarre de­ci­sion/​ac­tion al­gorithms are the way to win. For ex­am­ple, the world where the ex­tent to which your prefer­ences get satis­fied is de­ter­mined by what frac­tion of your skin is cov­ered in red body paint, etc, etc.

Also, there are other classes of wor­lds where I lose: for ex­am­ple, anti-in­duc­tive wor­lds. Why should I pay spe­cial at­ten­tion to the wor­lds that loosely obey the oc­cam/​com­plex­ity prior?

Per­haps I could frame it this way: the com­plex­ity prior is (in fact) coun­ter­in­tu­itive and alien to the hu­man mind. Why should I pay spe­cial at­ten­tion to wor­lds that con­form to it (sim­ple wor­lds)?

The an­swer I used to have was “be­cause it works”, which seemed to cache out as

“if I use a com­plex­ity prior to re­peat­edly make de­ci­sions, then my sub­jec­tive ex­pe­rience will be (mostly) of win­ning”

which I used to think was be­cause the Real world that we live in is, in fact, a sim­ple one, rather than a wish­ful-think­ing one, a red-body-paint one, or an anti-in­duc­tive one.

• It sounds like you’re as­sum­ing that peo­ple use a wish­ful-think­ing prior by de­fault, and have to be ar­gued into a com­plex­ity-based prior. This seems im­plau­si­ble to me.

I think the phe­nomenon of wish­ful think­ing doesn’t come from one’s prior, but from evolu­tion be­ing too stupid to de­sign a ra­tio­nal de­ci­sion pro­cess. That is, a part of my brain re­wards me for in­creas­ing the an­ti­ci­pa­tion of pos­i­tive fu­ture ex­pe­riences, even if that in­crease is caused by faulty rea­son­ing in­stead of good de­ci­sions. This causes me to en­gage in wish­ful think­ing (i.e., mis­calcu­lat­ing the im­pli­ca­tions of my prior) in or­der to in­crease my re­ward.

Per­haps I could frame it this way: the com­plex­ity prior is (in fact) coun­ter­in­tu­itive and alien to the hu­man mind.

I dis­pute this. Sure, some of the im­pli­ca­tions of the com­plex­ity prior are coun­ter­in­tu­itive, but it would be sur­pris­ing if none of them were. I mean, some the­o­rems of num­ber the­ory are coun­ter­in­tu­itive, but that doesn’t mean in­te­gers are aliens to the hu­man mind.

Why should I pay spe­cial at­ten­tion to wor­lds that con­form to it (sim­ple wor­lds)?

Sup­pose some­one gave you a wa­ter-tight ar­gu­ment that all pos­si­ble world are in fact real, and you have to make de­ci­sions based on which wor­lds you care more about. Would you re­ally adopt the “wish­ful-think­ing” prior and start putting all your money into lot­tery tick­ets or some­thing similar, or would your be­hav­ior be more or less un­af­fected? If it’s the lat­ter, don’t you already care more about wor­lds that are sim­ple?

“if I use a com­plex­ity prior to re­peat­edly make de­ci­sions, then my sub­jec­tive ex­pe­rience will be (mostly) of win­ning”

Per­haps this is just one of the ways an al­gorithm that cares about each world in pro­por­tion to its in­verse com­plex­ity could feel from the in­side?

• “if I use a com­plex­ity prior to re­peat­edly make de­ci­sions, then my sub­jec­tive ex­pe­rience will be (mostly) of win­ning”—Per­haps this is just one of the ways an al­gorithm that cares about each world in pro­por­tion to its in­verse com­plex­ity could feel from the in­side?

this is a good point, I’ll have to think about it.

• Sup­pose some­one gave you a wa­ter-tight ar­gu­ment that all pos­si­ble world are in fact real, and you have to make de­ci­sions based on which wor­lds you care more about. Would you re­ally adopt the “wish­ful-think­ing” prior and start putting all your money into lot­tery tick­ets or some­thing similar, or would your be­hav­ior be more or less un­af­fected?

I think that there would be a ques­tion about what “I” would ac­tu­ally ex­pe­rience.

There have been times in my younger days when I tried a bit of wish­ful think­ing—I think ev­ery­one has. Maybe, just maybe, if I wish hard enough for X, X will hap­pen? Well what you ac­tu­ally ex­pe­rience af­ter do­ing that is … failure. Wish­ing for some­thing doesn’t make it hap­pen—or if it does in some wor­lds, then I have ev­i­dence that I don’t in­habit those wor­lds.

So I sup­pose I am us­ing my mem­ory—which points to me hav­ing always been in a world that be­haves ex­actly as the com­plex­ity prior would pre­dict—as ev­i­dence that the thread of my sub­jec­tive ex­pe­rience will always be in a world that be­haves as the com­plex­ity prior would pre­dict, which is sort of like say­ing that only one par­tic­u­lar sim­ple world is real.

• You don’t be­lieve in af­fir­ma­tions? The self-help books about the power of pos­i­tive think­ing don’t work for you? What do you make of the fol­low­ing quote?

“Per­sonal op­ti­mism cor­re­lates strongly with self-es­teem, with psy­cholog­i­cal well-be­ing and with phys­i­cal and men­tal health. Op­ti­mism has been shown to be cor­re­lated with bet­ter im­mune sys­tems in healthy peo­ple who have been sub­jected to stress.”

• This is not the kind of wish­ful think­ing I was talk­ing about: I was talk­ing about wish­ing for \$1000 and it just ap­pear­ing in your bank ac­count.

• When craft­ing ones wishes, one should have at least some minor el­e­ment of re­al­ism.

If you pre­vi­ously did not bear such points in mind, you might want to con­sider re­vis­it­ing the tech­nique, to see if you can make some­thing of it. Un­less you figure you are already too op­ti­mistic, that is.

• You seem to be con­fus­ing plau­si­bil­ity with pos­si­bil­ity. The ex­is­tence of God seems plau­si­ble to many peo­ple, but whether or not the ex­is­tence of God is truly pos­si­ble is not clear. Rea­son­able peo­ple be­lieve that God is im­pos­si­ble, oth­ers that God is pos­si­ble, and oth­ers that God is nec­es­sary (i.e. God’s nonex­is­tance is im­pos­si­ble).

• Well, there are many weird and won­der­ful gods that are in­deed pos­si­ble, even if the par­tic­u­lar one that many peo­ple pro­fess to be­lieve in is self-con­tra­dic­tory, and there­fore in­co­her­ent.

• It wouldn’t quite throw all of our shit in the fan. If you know you’re liv­ing in a QM many wor­lds uni­verse you still have to op­ti­mize the borne prob­a­bil­ities, for ex­am­ple.

I think we can rule out the pop­u­lar re­li­gions as be­ing im­pos­si­ble wor­lds, but simu­lated wor­lds are pos­si­ble wor­lds, and in some sub­set of them, you can know this.

In the one’s where you can know differ­en­ti­ate to some de­gree, there are cer­tainly ac­tions that one could take to help his ‘simu­lated’ selves at the cost of the ‘non­si­mu­lated’ selves, if you cared.

I guess the ques­tion is of whether it’s even con­sis­tent to care about be­ing “simu­lated” or not, and where you draw the line (what if you have some in­for­ma­tion rate in from the out­side and have some in­fluence over it? What if its the ex­act same hard­ware just pluggged in like in ‘the ma­trix’?)

My guess is that it is gonna turn out to not make any sense to care about them differ­ently, and that theres some nat­u­ral weight­ing which we haven’t yet figured out. Maybe weight each copy by the re­dun­dancy in the pro­ces­sor (eg if each tran­sis­tor is X atoms big, then that can be thought of X copies liv­ing in the same house) or by the power they have to in­fluence the world, or some­thing. Both of those have prob­lems, but I can’t think of any­thing bet­ter.

• I think we can rule out the pop­u­lar re­li­gions as be­ing im­pos­si­ble worlds

There are pos­si­ble wor­lds that are pretty good ap­prox­i­ma­tions to pop­u­lar re­li­gions.

If you know you’re liv­ing in a QM many wor­lds uni­verse you still have to op­ti­mize the borne probabilities

I don’t un­der­stand this…

• There are pos­si­ble wor­lds that are pretty good ap­prox­i­ma­tions to pop­u­lar re­li­gions.

True...

I don’t un­der­stand this...

The pa­per does a much more thor­ough job than I, but the sum­mary is that the only con­sis­tent way to carve is into borne prob­a­bil­ities, so you have to weight branches ac­cord­ingly. I think this has to due with the am­pli­tude squared be­ing con­served, so that the eb­bo­ri­ans equiv­a­lent would be their thick­ness, but I ad­mit some con­fu­sion here.

This means there’s at least some sense of prob­a­bil­ity in which you don’t get to ‘wish away’, though it’s still pos­si­ble to only care about wor­lds where “X” is true (though in gen­eral you ac­tu­ally do care about the other wor­lds)

• There are plenty of pos­si­ble wor­lds (in­finitely many of them) where quan­tum me­chan­ics is false; so I don’t see how this helps.

• It means that if you are in one, prob­a­bil­ity does not come down to only prefer­ences. I sup­pose that since you can never be ab­solutely sure you’re in one, you still have to find out your weight­ings be­tween wor­lds where there might be noth­ing but prefer­ences.

The other point is that I se­ri­ously doubt there’s any­thing built into you that makes you not care about pos­si­ble wor­lds where QM is true, so even if it does come down to ‘mere prefer­ences’, you can still make mis­takes.

The ex­is­tence of an ob­jec­tive weight­ing scheme within one set of pos­si­ble wor­lds gives me some hope of an ob­jec­tive weight­ing be­tween all pos­si­ble wor­lds, but note all that much, and it’s not clear to me what that would be. Maybe the set of all pos­si­ble wor­lds is countable, and each world is weighted equally?

• Maybe the set of all pos­si­ble wor­lds is countable, and each world is weighted equally?

I am not re­ally sure what to make of weight­ings on pos­si­ble wor­lds. Over­all, on this is­sue, I think I am go­ing to have to ad­mit that I am thor­oughly con­fused.

By the way, do you mean “finite” here, rather than countable?

• Yeah, but the con­fu­sion gets bet­ter as the wor­lds be­come more similar. How to weight be­tween QM wor­lds and nonQM wor­lds is some­thing I haven’t even seen an at­tempt to ex­plain, but how to weight within QM wor­lds has been ex­plained, and how to weight in the sleep­ing beauty prob­lem is quite straight for­ward.

I meant countable, but now that you men­tion it I think I should have said finite- I’ll have to think about this some more.

• Lump­ing prob­a­bil­ities in with util­ities sounds pretty close to Vladimir Nesov’s Rep­re­sent­ing Prefer­ence by Prob­a­bil­ity Mea­sures.

• Be­fore I’ve ob­served any­thing, there seems to be no rea­son to be­lieve that I’m more likely to be in one world than an­other, but we can’t let all their weights be equal.

We can’t? Why not? Es­ti­mat­ing the prob­a­bil­ity of two heads on two coin­flips as 25% is giv­ing ex­is­tence in wor­lds with heads-heads, heads-tails, tails-heads, and tails-tails equal weight. The same is true of a more com­pli­cated propo­si­tion like “There is a low prob­a­bil­ity that Bigfoot ex­ists”—giv­ing ev­ery pos­si­ble ar­range­ment of ob­jects/​atoms/​in­for­ma­tion equal weight, and then rul­ing out the ones that don’t re­sult in the ev­i­dence we’ve ob­served, few of these wor­lds con­tain Bigfoot.

• giv­ing ev­ery pos­si­ble ar­range­ment of ob­jects/​atoms/​in­for­ma­tion equal weight

Without an ar­bi­trary up­per bound on com­plex­ity, there are in­finitely many pos­si­ble ar­range­ments.

• The­o­ret­i­cally, it’s not in­finite be­cause of the gran­u­lar­ity of time/​space, speed of light, and so on.

Prac­ti­cally, we can get around this be­cause we only care about a tiny frac­tion of the pos­si­ble vari­a­tion in ar­range­ments of the uni­verse. In a coin flip, we only care about whether a coin is heads-up or tails-up, not the en­ergy state of ev­ery sub­atomic par­ti­cle in the coin.

This mat­ters in the case of a bi­ased coin—let’s say bi­ased to­wards heads 66%. This, I think, is what Wei meant when he said we couldn’t just give equal weights to all pos­si­ble uni­verses—the ones where the coin lands on heads and the ones where it lands on tails. But I think “uni­verses where the coin lands on heads” and “uni­verses where the coin lands on tails” are un­nat­u­ral cat­e­gories.

Con­sider how the prob­a­bil­ity of win­ning the lot­tery isn’t .5 be­cause we choose with equal weight be­tween the two al­ter­na­tives”I win” and “I don’t win”. Those are un­nat­u­ral cat­e­gories, and in­stead we need to choose with equal weight be­tween “I win”, “John Q. Smith of Lit­tle Rock Arkansas wins”, “Mary Brown of San An­to­nio, Texas, wins” and so on to mil­lions of other peo­ple. The un­nat­u­ral cat­e­gory “I don’t win” con­tains mil­lions of more nat­u­ral cat­e­gories.

So on the bi­ased coin flip, the cat­e­gories “the coin lands heads” and “the coin lands tails” con­tains a bunch of cat­e­gories of lower-level events about col­li­sions of air molecules and coin molecules and amounts of force one can use to flip a coin, and two-thirds of those events are in the “coin lands heads” cat­e­gory. But among those lower-level events, you choose with equal weight.

True, be­neath these lower-level cat­e­gories about col­li­sions of air molecules, there are prob­a­bly even lower things like vibra­tions of su­per­strings or bits in the world-simu­la­tion or what­ever the low­est level of re­al­ity is, but as long as these be­have math­e­mat­i­cally I don’t see why they pre­vent us from bas­ing a the­ory of prob­a­bil­ity on the effects of low level con­di­tions.

• The­o­ret­i­cally, it’s not in­finite be­cause of the gran­u­lar­ity of time/​space, speed of light, and so on.

Th­ese ini­tial weights are sup­posed to be as­signed be­fore tak­ing into ac­count any­thing you have ob­served. But even now (un­der the sec­ond in­ter­pre­ta­tion in my list) you can’t be sure that the world you’re in is finite. So, sup­pose there is one pos­si­ble world for each in­te­ger in the set of all in­te­gers, or one pos­si­ble world for each set in the class of all sets. How could one as­sign equal weight to all pos­si­ble wor­lds, and have the weights add up to 1?

Prac­ti­cally, we can get around this be­cause we only care about a tiny frac­tion of the pos­si­ble vari­a­tion in ar­range­ments of the uni­verse. In a coin flip, we only care about whether a coin is heads-up or tails-up, not the en­ergy state of ev­ery sub­atomic par­ti­cle in the coin.

I don’t think that gets around the prob­lem, be­cause there is an in­finite num­ber of pos­si­ble wor­lds where the en­ergy state of nearly ev­ery sub­atomic par­ti­cle en­codes some valuable in­for­ma­tion.

• How could one as­sign equal weight to all pos­si­ble wor­lds, and have the weights add up to 1?

By the same method we do calcu­lus. In­stead of sum of the pos­si­ble wor­lds we in­te­grate over the pos­si­ble wor­lds (which is a in­finite sum of in­finites­i­mally small val­ues). For ex­plicit con­struc­tion on how this is done any ba­sic calcu­lus book is enough.

• My un­der­stand­ing is that it’s pos­si­ble to have a uniform dis­tri­bu­tion over a finite set, or an in­ter­val of the re­als, but not over all in­te­gers, or all re­als, which is why I said in the sen­tence be­fore the one you quotes, “sup­pose there is one pos­si­ble world for each in­te­ger in the set of all in­te­gers.”

• There is a 1:1 map­ping be­tween “the set of re­als in [0,1]” and “the set of all re­als”. So take your uniform dis­tri­bu­tion on [0,1] and put it through such a map­ping… and the re­sult is non-uniform. Which pretty much kills the idea of “uniform ⇔ each el­e­ment has the same prob­a­bil­ity as each other”.

There is no such thing as a con­tin­u­ous dis­tri­bu­tion on a set alone, it has to be on a met­ric space. Even if you make a met­ric space out of the set of all pos­si­ble uni­verses, that doesn’t give you a uni­ver­sal prior, be­cause you have to choose what met­ric it should be uniform with re­spect to.

(Can you have a uniform “con­tin­u­ous” dis­tri­bu­tion with­out a con­tinuum? The ra­tio­nals in [0,1]?)

• As there is the 1:1 map­ping be­tween set of all re­als and unit in­ter­val we can just use the unit in­ter­val and define a uniform map­ping there. As what­ever dis­tri­bu­tion you choose we can map it into unit in­ter­val as Peng­vado said.

In case of set of all in­te­gers I’m not com­pletely cer­tain. But I’d look at the set of com­putable re­als which we can use for much of math­e­mat­ics. Nor­mal calcu­lus can be done with just com­putable re­als (set of all num­bers where there is an al­gorithm which pro­vides ar­bi­trary dec­i­mal in a finite time). So ba­si­cally we have a map­ping from com­putable re­als on unit in­ter­val into set of all in­te­gers.

Another ques­tion is that is the uniform dis­tri­bu­tion the en­tropy max­imis­ing dis­tri­bu­tion when we con­sider set of all in­te­gers?

From a phys­i­cal stand­point why are you in­ter­ested in countably in­finite prob­a­bil­ity dis­tri­bu­tions? If we as­sume dis­crete phys­i­cal laws we’d have finite amount of pos­si­ble wor­lds, on the other hand if we as­sume con­tin­u­ous we’d have un­countably in­finite amount which can be mapped into unit in­ter­val.

From the top of my head I can imag­ine set of dis­crete wor­lds of all sizes which would be countably in­finite. What other kinds of wor­lds there could be where this would be rele­vant?

• The­o­ret­i­cally, it’s not in­finite be­cause of the gran­u­lar­ity of time/​space, speed of light, and so on.

(Nit­pick: Space­time isn’t quan­tized AFAIK in stan­dard physics, and then there are still con­tin­u­ous quan­tum am­pli­tudes.)

This, I think, is what Wei meant when he said we couldn’t just give equal weights to all pos­si­ble uni­verses—the ones where the coin lands on heads and the ones where it lands on tails. But I think “uni­verses where the coin lands on heads” and “uni­verses where the coin lands on tails” are un­nat­u­ral cat­e­gories.

I thought Wei was talk­ing about sin­gle wor­lds (what­ever those may be), not sets of wor­lds. Ap­plied to sets of wor­lds, this seems cor­rect.

• Yvain said the finite­ness well, but I think the “in­finitely many pos­si­ble ar­range­ments” needs a lit­tle elab­o­ra­tion.

In any con­tin­u­ous prob­a­bil­ity dis­tri­bu­tions we have in­finitely many (ac­tu­ally un­countably in­finitely many) pos­si­bil­ities, and this makes the prob­a­bil­ity of any sin­gle out­come 0. Which is the rea­son why, in the case of con­tin­u­ous dis­tri­bu­tions, we talk about prob­a­bil­ity of the out­come be­ing on a cer­tain in­ter­val (a col­lec­tion of in­finitely many ar­range­ments).

So in­stead of count­ing the in­di­vi­d­ual ar­range­ments we calcu­late in­te­grals over some set of ar­range­ments. In­finitely many ar­range­ments is no hin­drance to ap­ply­ing prob­a­bil­ity the­ory. Ac­tu­ally if we can as­sume con­tin­u­ous dis­tri­bu­tion it makes some things much eas­ier.

• Good point. Does this work over all in­finite sets, though? In­te­gers? Ra­tion­als?

• It does work, ac­tu­ally if we’re us­ing In­te­gers (there are as many in­te­gers as Ra­tion­als so we don’t need to care about the lat­ter set) we get the good old dis­crete prob­a­bil­ity dis­tri­bu­tion where we ei­ther have finite num­ber of pos­si­bil­ities or at most countable in­finity of pos­si­bil­ities, e.g set of all In­te­gers.

Real num­bers are strictly larger set than in­te­gers, so in con­tin­u­ous dis­tri­bu­tion we have in a sense more pos­si­bil­ities than countably in­finite dis­crete dis­tri­bu­tion.

• Hmmm—car­ing as a part of re­al­ity? Why not just flip things up, and con­sider that emo­tion is also part of re­al­ity. Ran­dom by any other name. Try to ex­clude it and you’ll find you can’t no mat­ter how in­finitely many wor­lds you sup­pose. There’s also calcu­lus to ir­ra­tional­ity . . .

• The “car­ing” in­ter­pre­ta­tion doesn’t say that car­ing is part of re­al­ity (ex­cept in­so­far as minds are im­ple­mented in re­al­ity). Rather, it says that prob­a­bil­ity isn’t part of re­al­ity, it’s part of de­ci­sion the­ory (again ex­cept in­so­far as minds are im­ple­mented in re­al­ity).

• cool! but can you re­ally posit ar­tifi­cial in­tel­li­gence (de­ci­sion the­ory has to get en­acted some­where) and not al­low mind as part of re­al­ity?

• All pos­si­ble wor­lds are real, and prob­a­bil­ities rep­re­sent how much I care about each world. … Which wor­lds I care more or less about seems ar­bi­trary.

This view seems ap­peal­ing to me, be­cause 1) de­cid­ing that all pos­si­ble wor­lds are real seems to fol­low from the Coper­ni­can prin­ci­ple, and 2) if all wor­lds are real from the per­spec­tive of their ob­servers, as you said it seems ar­bi­trary to say which wor­lds are more real.

But on this view, what do I do with the ob­served fre­quen­cies of past events? When­ever I’ve flipped a coin, heads has come up about half the time. If I ac­cept op­tion 4, am I giv­ing up on the idea that these reg­u­lar­i­ties mean any­thing?

• 26 Nov 2011 18:11 UTC
0 points

What does real even mean, by the way? In­ter­pre­ta­tion 1 with real taken to mean ‘of or per­tain­ing to the world I’m in’ (as I would) is equiv­a­lent to In­ter­pre­ta­tion 2 with real taken to mean ‘pos­si­ble’ (as Teg­mark would, IIUC) and to In­ter­pre­ta­tion 3 with real taken to mean ‘likely’ and to In­ter­pre­ta­tion 4 with real taken to mean ‘im­por­tant to me’.

• It de­pends. We use the term “prob­a­bil­ity” to cover a va­ri­ety of differ­ent things, which can be han­dled by similar math­e­mat­ics but are not the same.

For ex­am­ple, sup­pose that I’m play­ing black­jack. Given a cer­tain dis­po­si­tion of cards, I can calcu­late a prob­a­bil­ity that ask­ing for the next card will bust me. In this case the state of the world is fixed, and prob­a­bil­ity mea­sures my ig­no­rance. The fact that I don’t know which card would be dealt to me doesn’t change the fact that there’s a spe­cific card on the top of the deck wait­ing to be dealt. If I knew more about the situ­a­tion (per­haps by count­ing cards) I might have a bet­ter idea of which cards could pos­si­bly be on top of the deck, but the same card would still be on top of the deck. In this situ­a­tion, case 1 ap­plies from the choices above.

Alter­nately con­sider pho­tons go­ing through a dou­ble slit in the clas­si­cal quan­tum physics ex­per­i­ment. If the holes are of equal size and ge­om­e­try, a pho­ton has a 50% chance of pass­ing through each slit (the prob­a­bil­ities can be ad­justed, for ex­am­ple by chang­ing the width of one slit). One of the ba­sic re­sults of quan­tum physics is that the pro­file of the light through both slits is not the same as the sum of the pro­files of the light through each slit. In gen­eral, it is not pos­si­ble to say which slit a given pho­ton when through, and at­tempt­ing to make that mea­sure­ment changes the an­swer. In this situ­a­tion, case 3 of the above post seems to ap­ply.

My point is that the post’s ques­tion can’t be an­swered for prob­a­bil­ities in gen­eral. It de­pends.

• 2 and 4 are much the same if you only care about wor­lds you are in.

• The post would be much bet­ter if a defi­ni­tion of “pos­si­ble world” was given. When giv­ing defi­ni­tions, per­haps to define what does “real” pre­cisely mean would be benefi­cial.

More or less, I in­ter­pret “re­al­ity” as all things which can be ob­served. “Pos­si­ble”, in my lan­guage”, is some­thing which I can imag­ine and which doesn’t con­tra­dict facts that I already know. This is some­what sub­jec­tive defi­ni­tion, but pos­si­bil­ity ob­vi­ously de­pends sub­jec­tive knowl­edge. I have flipped a coin. Be­fore I have looked at the re­sult, it was pos­si­ble that it came up heads. After I have looked at it, it’s clear that it came up tails, heads are im­pos­si­ble.

Need­less to say, peo­ple rarely imag­ine whole wor­lds. Rather, they use the word “pos­si­ble” when spec­u­lat­ing about un­know parts of this world. Which may be con­fus­ing, since our in­tu­itive un­der­stand­ing of the word doesn’t match its use.

Even if defined some­how ob­jec­tively (as e.g. pos­si­ble world is any world iso­mor­phic to a for­mal sys­tem with prop­er­ties X), it seems al­most ob­vi­ous that real world(s) and pos­si­ble wor­lds are differ­ent cat­e­gories. If not, there is no need to have dis­tinct names for them.

So be­fore cre­at­ing the­o­ries about what prob­a­bil­ity means, I sug­gest we unite the lan­guage. Th­ese things have been dis­cussed here already sev­eral times, but I don’t think there is a con­sen­sus in in­ter­pre­ta­tion of “pos­si­ble”, “real”, “world”, “ar­bi­trary”. And, af­ter all, I am not sure whether “prob­a­bil­ity” even should be in­ter­preted us­ing these terms. It al­most feels like “prob­a­bil­ity” is a more fun­da­men­tal term than “pos­si­ble” or “ar­bi­trary”.

I must ad­mit that I am bi­ased against “pos­si­ble wor­lds” and similar phrases, be­cause they tend to ap­pear mostly in the­olog­i­cal and philo­soph­i­cal dis­cus­sions, whose rather empty con­clu­sions are dis­satis­fac­tory. I am afraid of lack of guidelines strong enough to keep think­ing in limits of ra­tio­nal­ity.

• Why should prob­a­bil­ities mean any­thing? How how would you be­have differ­ently if you de­cided (or learned) a given in­ter­pre­ta­tion was cor­rect?

As long as there’s no differ­ence, and your ac­tions add up to nor­mal­ity un­der any of the in­ter­pre­ta­tions, then I don’t see why an in­ter­pre­ta­tion is needed at all.

• The differ­ent in­ter­pre­ta­tions sug­gest differ­ent ap­proaches to an­swer the ques­tion of “what is the right prior?” and also differ­ent ap­proaches to de­ci­sion the­ory. I men­tioned that the “car­ing” in­ter­pre­ta­tion fits well with UDT.

• Can’t you choose your (ara­tional) prefer­ences to get any be­havi­our (de­ci­sion the­ory) no mat­ter what in­ter­pre­ta­tion you choose?

• Prefer­ences may be ara­tional, but they’re not com­pletely ar­bi­trary. In moral philos­o­phy there are still ar­gu­ments for what one’s prefer­ences should be, even if they are gen­er­ally much weaker than the ar­gu­ments in ra­tio­nal­ity. Differ­ent in­ter­pre­ta­tions in­fluence what kinds of ar­gu­ments ap­ply or make sense to you, and there­fore in­fluence your prefer­ences.

• How can there be ar­gu­ments about what prefer­ences should be? Aren’t they, well, a sort of un­moved mover, a pri­mal cause? (To use some erst­while philo­soph­i­cal terms :-)

I can un­der­stand meta-ar­gu­ments that say your prefer­ences should be con­sis­tent in some sense, or that ar­gue about sub­goal prefer­ences given some su­per­goals. But even un­der strict con­straints of that kind, you have a lot of lat­i­tude, from hu­mans to pa­per­clip max­i­miz­ers on out. Within that range, does in­ter­pret­ing prob­a­bil­ities differ­ently re­ally give you ex­tra power you can’t get by fine­tun­ing your prefs?

Edit: the rea­son I’d perfer edit­ing prefs is that talk­ing about the Mean­ing of Prob­a­bil­ities sets off my ma­te­ri­al­ism sen­sors. It leads to things like mul­ti­ple-world the­o­ries be­cause they’re easy to think about as an in­etr­pre­ta­tion of QM, re­gard­less of whether they ac­tu­ally ex­ist. Then they can ac­tu­ally nega­tively af­fect our prefs or be­hav­ior.

• Re: “How can there be ar­gu­ments about what prefer­ences should be?”

The idea that some prefer­ences are “bet­ter” than other ones is known as “moral re­al­ism”.

• Wikipe­dia says moral re­al­ists (in gen­eral) claim that moral propo­si­tions can be true or false as ob­jec­tive facts but their truth can­not be ob­served or ver­ified. This doesn’t make any sense. Sounds like re­li­gion.

• Are you look­ing at http://​​en.wikipe­dia.org/​​wiki/​​Mo­ral_re­al­ism …?

Care to quote an offend­ing sec­tion about moral truths not be­ing ob­serverv­able or ver­ifi­able?

• Un­der the sec­tion “Crit­i­cisms”:

Others are crit­i­cal of moral re­al­ism be­cause it pos­tu­lates the ex­is­tence of a kind of “moral fact” which is non­ma­te­rial and does not ap­pear to be ac­cessible to the sci­en­tific method. Mo­ral truths can­not be ob­served in the same way as ma­te­rial facts (which are ob­jec­tive), so it seems odd to count them in the same cat­e­gory. One emo­tivist coun­ter­ar­gu­ment (al­though emo­tivism is usu­ally non-cog­ni­tivist) alleges that “wrong” ac­tions pro­duce mea­surable re­sults in the form of nega­tive emo­tional re­ac­tions, ei­ther within the in­di­vi­d­ual trans­gres­sor, within the per­son or peo­ple most di­rectly af­fected by the act, or within a (prefer­ably wide) con­sen­sus of di­rect or in­di­rect ob­servers.

Re­gard­ing the emo­tivist crit­i­cism, it begs a lot of ques­tions. Surely not all nega­tive emo­tional re­ac­tions sig­nal wrong moral ac­tions. Be­sides, emo­tivism isn’t al­igned with moral re­al­ism.

• I see—thanks.

That some crit­i­cisms of moral re­al­ism ap­pear to lack co­her­ence does not seem to me to be a point that counts against the idea.

I ex­pect moral re­al­ists would deny that moral­ity is any more non­ma­te­rial than any other kind of in­for­ma­tion—and would also deny that it does not ap­pear to be ac­cessible to the sci­en­tific method.

• If moral re­al­ism acts as a sys­tem of log­i­cal propo­si­tions and de­duc­tions, then it has to have moral ax­ioms. How are these grounded in ma­te­rial re­al­ity? How can they be any­thing more than “be­cause i said so and I hope you’ll agree”? Isn’t the choice of ax­ioms done us­ing a moral the­ory nom­i­nally op­posed to moral re­al­ism, such as emo­tivism, or (amoral) util­i­tar­i­anism?

• One way would be to con­sider the fu­ture of civ­i­liza­tion. At the mo­ment, we ob­serve a Shift­ing Mo­ral Zeit­geist. How­ever, in the fu­ture we may see ideas about how to be­have to­wards other agents set­tle down into an op­ti­mal re­gion. If that turns out to be a global op­ti­mum—rather than a lo­cal one—i.e. much the same rules would be found by most sur­viv­ing aliens—then that would rep­re­sent a good foun­da­tion for the ideas of moral re­al­ism.

Even to­day, it should be pretty ob­vi­ous that some moral sys­tems are “bet­ter” than oth­ers (“bet­ter” in the sense of pro­mot­ing the sur­vival of those sys­tems). That doesn’t nec­es­sar­ily mean there’s a “best” one—but it leaves that pos­si­bil­ity open.

• It might also sound like sci­ence—don’t sci­en­tists gen­er­ally claim that propo­si­tions about the world can be true or false, but can­not be di­rectly ob­served or ver­ified?

Joshua Greene’s the­sis “The Ter­rible, Hor­rible, No Good, Very Bad Truth about Mo­ral­ity and What to Do About it” might be a de­cent in­tro­duc­tion to moral re­al­ism /​ ir­re­al­ism. Over­all it is an ar­gu­ment for ir­re­al­ism.

• In sci­ence, a propo­si­tion about the world can gen­er­ally be proven or dis­proven with ar­bi­trary prob­a­bil­ity, so you can be­come as sure about it as you like if you in­vest enough re­sources.

In moral re­al­ism, propo­si­tions are purely log­i­cal con­structs, and can be proven true or false just like a math­e­mat­ica propo­si­tion. Their truth is one with the truth of the ax­ioms used, and the ax­ioms can’t be proven or dis­proven with any de­gree of cer­tainty; they are sim­ply ac­cepted or not ac­cepted. The moral­ity is in­ter­nally con­sis­tent, but you can’t de­rive it from the real world, and you can’t de­rive any fact about the real world from the moral­ity. That sounds just like the­ol­ogy to me. (The differ­ence be­tween this and or­di­nary math or logic, is that math­e­mat­i­cal con­structs aren’t sup­posed to lead to should or ought state­ments about be­hav­ior.)

I will read Greene’s the­sis, but as far as I can tell it ar­gues against moral re­al­ism (and does it well), so it won’t help me un­der­stand why any­one would be­lieve in it.

• How can there be ar­gu­ments about what prefer­ences should be?

Well, I don’t know what many of my prefer­ences should be. How can I find out ex­cept by look­ing for and listen­ing to ar­gu­ments?

Aren’t they, well, a sort of un­moved mover, a pri­mal cause? (To use some erst­while philo­soph­i­cal terms :-)

No, not for hu­mans any­way.

• Well, I don’t know what many of my prefer­ences should be. How can I find out ex­cept by look­ing for and listen­ing to ar­gu­ments?

That im­plies there’s some ob­jec­tively-defin­able stan­dard for prefer­ences which you’ll be able to rec­og­nize once you see it. Also, it begs the ques­tion of what in your cur­rent prefer­ences says “I have to go out and get some more/​differ­ent prefer­ences!” From a goal-driven in­tel­li­gence’s POV, ask­ing oth­ers to mod­ify your prefs in un­speci­fied ways is pretty much the anti-ra­tio­nal act.

• I think we need to dis­t­in­guish be­tween what a ra­tio­nal agent should do, and what a non-ra­tio­nal hu­man should do to be­come more ra­tio­nal. Nesov’s re­ply to you also con­cerns the former, I think, but I’m more in­ter­ested in the lat­ter here.

Un­like a ra­tio­nal agent, we don’t have well-defined prefer­ences, and the prefer­ences that we think we have can be changed by ar­gu­ments. What to do about this situ­a­tion? Should we stop think­ing up or listen­ing to ar­gu­ments, and just fill in the fuzzy parts of our prefer­ences with ran­dom­ness or in­differ­ence, in or­der to em­u­late a ra­tio­nal agent in the most di­rect man­ner pos­si­ble? That doesn’t make much sense to me.

I’m not sure what we should do ex­actly, but what­ever it is, it seems like ar­gu­ments must make up a large part of it.

• Please see my re­ply to Nesov above, too.

I think we shouldn’t try to em­u­late ra­tio­nal agents at all, in the sense that we shouldn’t pre­tend to have ra­tio­nal­ity-style prefer­ences and su­per­goals; as a mat­ter of fact we don’t have them.

Up to here we seem to agree, we just use differ­ent ter­minol­ogy. I just don’t want to con­flate ra­tio­nal prefer­ences with hu­man prefer­ences be­cause they the two sys­tems be­have very differ­ently.

Just as an ex­am­ple, in sig­nal­ling the­o­ries of be­havi­our, you may con­sciously be­lieve that your prefer­ences are very differ­ent from what your be­havi­our is ac­tu­ally op­ti­miz­ing for when noone is look­ing. A ra­tio­nal agent wouldn’t nor­mally have sep­a­rate con­scious/​un­con­scious minds un­less only the con­scious part was sbu­ject to out­side in­spec­tion. In this ex­am­ple, it makes sense to up­date sig­nal­ling-prefer­ences some­times, be­cause they’re not your ac­tual act­ing-prefer­ences.

But if you con­sciously in­tend to act out your (con­scious) prefer­ences, and also in­tend to keep chang­ing them in not-always-fore­see­able ways, then that isn’t ra­tio­nal­ity, and when there could be con­fu­sion due to con­text (such as on LW most of the time) I’d pre­fer not to use the term “prefer­ences” about hu­mans, or to make clear what is meant.

• That ar­gu­ments mod­ify prefer­ence means that you are (de­no­ta­tion­ally) ar­riv­ing at differ­ent prefer­ences de­pend­ing on ar­gu­ments. This means that, from the per­spec­tive of a spe­cific given prefer­ence (or “true” neu­tral prefer­ence not bi­ased by spe­cific ar­gu­ments), you fail to ob­tain op­ti­mal ra­tio­nal de­ci­sion al­gorithm, and thus to achieve high-prefer­ence strat­egy. But at the same time, “ab­sence of ac­tion” is also an ac­tion, so not ex­plor­ing the ar­gu­ments may as well be a worse choice, since you won’t be mov­ing for­ward to­wards more clear un­der­stand­ing of your own prefer­ence, even if the prefer­ence that you are go­ing to un­der­stand will be some­what bi­ased com­pared to the un­known origi­nal one.

• Ir­ra­tional per­cep­tion of ar­gu­ments leads to mod­ifi­ca­tion of prefer­ence, which is bad for origi­nal prefer­ence, but

• Con­sid­er­ing moral ar­gu­ments leads to a more clear un­der­stand­ing of some prefer­ence close to the origi­nal one, which al­lows to make more ra­tio­nal de­ci­sions, which is good for the origi­nal prefer­ence.

• FWIW, my prefer­ences have not been changed by ar­gu­ments in the last 20 years. So I don’t think your “we” in­cludes me.

• As an ex­am­ple, con­sider the ar­gu­ments in form of proofs/​dis­proofs of the state­ments that you are in­ter­ested in. In­for­ma­tion doesn’t nec­es­sar­ily “change” or “de­ter­mine ar­bi­trar­ily” the things you take from it, it may help you to com­pute an ob­ject in which you are already in­ter­ested, with­out chang­ing that ob­ject, and at the same time be es­sen­tial in mov­ing for­ward. If you have an al­gorithm, it doesn’t mean that you know what this al­gorithm will give you in the end, what the al­gorithm “means”. Re­sist the illu­sion of trans­parency.

• I don’t un­der­stand what you’re say­ing as ap­plied to this ar­gu­ment. That Wei Dai has an al­gorithm for mod­ify­ing his prefer­ences and he doesn’t know what the end out­put of that al­gorithm will be?

• There will always be some­thing about prefer­ence that you don’t know, and it’s not the ques­tion of mod­ify­ing prefer­ence, it’s a ques­tion of figur­ing out what the fixed un­mod­ifi­able prefer­ence im­plies. Mod­ify­ing prefer­ence is ex­actly the wrong way of go­ing about this.

If we figure out the con­cep­tual is­sues of FAI, we’d ba­si­cally have the al­gorithm that is our prefer­ences, but not in in­finite and un­know­able nor­mal “ex­e­cu­tion trace” de­no­ta­tional “form”.

• As Wei says be­low, we should con­sider ra­tio­nal agents (who have ex­plicit prefer­ences sep­a­rate from the rest of their cog­ni­tive ar­chi­tec­ture) sep­a­rately from hu­mans who want to ap­prox­i­mate that in some ways.

I think that if we first define sep­a­rate prefer­ences, and then pro­ceed to mod­ify them over and over again, this is so differ­ent from ra­tio­nal agents that we shouldn’t call it prefer­ences at all. We can talk about e.g. morals in­stead, or about habits, or bi­ases.

On the other hand if we define hu­man prefer­ences as ‘what­ever hu­man be­hav­ior hap­pens to op­ti­mize’, then there’s noth­ing in­ter­est­ing about chang­ing our prefer­ences, this is some­thing that hap­pens all the time whether we want it to or not. Un­der this defi­ni­tion Wei’s state­ment that he de­liber­ately makes it hap­pen is un­clear (the to­tal­ity of a hu­man’s be­havi­our, knowl­edge, etc. is sub­tly chang­ing over time in any case) so I as­sumed he was us­ing the former defi­ni­tion.

• There is no clear-cut di­chotomy be­tween defin­ing some­thing com­pletely at the be­gin­ning and do­ing things ar­bi­trar­ily as we go. In­stead of defin­ing prefer­ence for ra­tio­nal agents, in a com­plete, finished form, and then see­ing what hap­pens, con­sider a pro­cess of figur­ing out what prefer­ence is. This is nei­ther a way to ar­rive at the fi­nal an­swer, at any point, nor a his­tory of ob­serv­ing of “what­ever hap­pens”. Ra­tional agent is an im­pos­si­ble con­struct, but some­thing ir­ra­tional agents as­pire to be, never ob­tain­ing. What they want to be­come isn’t di­rectly re­lated to what they “ap­pear” to strive to­wards.

• I un­der­stand. So you’re say­ing we should in­deed use the term ‘prefer­ence’ for hu­mans (and a lot of other agents) be­cause no re­ally ra­tio­nal agents can ex­ist.

Ac­tu­ally, why is this true? I don’t know about perfect ra­tio­nal­ity, but why shouldn’t an agent ex­ist whose prefer­ences are com­pletely speci­fied and un­chang­ing?

• I un­der­stand. So you’re say­ing we should in­deed use the term ‘prefer­ence’ for hu­mans (and a lot of other agents) be­cause no re­ally ra­tio­nal agents can ex­ist.

Right. Ex­cept that re­ally ra­tio­nal agents might ex­ist, but not if their prefer­ences are pow­er­ful enough, as hu­mans’ have ev­ery chance to be. And what­ever we ir­ra­tional hu­mans, or our godlike but still, strictly speak­ing, ir­ra­tional FAI try to do, the con­cept of “prefer­ence” still needs to be there.

Ac­tu­ally, why is this true? I don’t know about perfect ra­tio­nal­ity, but why shouldn’t an agent ex­ist whose prefer­ences are com­pletely speci­fied and un­chang­ing?

Again, it’s not about chang­ing prefer­ence. See these com­ments.

An agent can have a com­pletely speci­fied and un­chang­ing prefer­ence, but still not know ev­ery­thing about it (and never able to know ev­ery­thing about it). In par­tic­u­lar, this is a con­se­quence of halt­ing prob­lem: if you have source code of a pro­gram, this code com­pletely speci­fies whether this pro­gram halts, and you may run this code for ar­bi­trar­ily long time with­out ever chang­ing it, but still not know whether it halts, and not be­ing able to ever figure that out, un­less you are lucky to ar­rive at a solu­tion in this par­tic­u­lar case.

• OK, I un­der­stand now what you’re say­ing. I think the main differ­ence, then, be­tween prefer­ences in hu­mans and in perfect (the­o­ret­i­cal) agents is that our prefer­ences aren’t sep­a­rate from the rest of our mind.

• I think the main differ­ence, then, be­tween prefer­ences in hu­mans and in perfect (the­o­ret­i­cal) agents is that our prefer­ences aren’t sep­a­rate from the rest of our mind.

I don’t un­der­stand this point.

• Ra­tional (de­signed) agents can have an ar­chi­tec­ture with prefer­ences (de­ci­sion mak­ing parts) sep­a­rate from other pieces of their minds (mem­ory, calcu­la­tions, plan­ning, etc.) Then it’s easy (well, eas­ier) to rea­son about chang­ing their prefer­ences be­cause we can hold the other parts con­stant. We can ask things like “given what this agent knows, how would it be­have un­der prefer­ence sys­tem X”?

The agent may also be able to simu­late pro­posed mod­ifi­ca­tions to its prefer­ences with­out hav­ing to simu­late its en­tire mind (which would be ex­pen­sive). And, in­deed, a suffi­ciently sim­ple prefer­ence sys­tem may be cho­sen so that it is not sub­ject to the halt­ing prob­lem and can be rea­soned about.

In hu­mans though, prefer­ences and ev­ery other part of our minds in­fluence one an­other. While I’m hold­ing a philo­soph­i­cal dis­cus­sion about moral­ity and de­cid­ing how to up­date my so-called prefer­ences, my de­ci­sions hap­pen to be af­fected by hunger or tired­ness or re­mem­ber­ing hav­ing had good sex last night. There are lots of bi­ases that are not per­ceived di­rectly. We can’t make ra­tio­nal de­ci­sions eas­ily.

In ra­tio­nal agents who are self-mod­ify­ing prefer­ences, the new prefs are de­ter­mined by the old prefs, i.e. via sec­ond-or­der prefs. But in hu­mans prefs are po­ten­tially de­ter­mined by the en­tire state of mind, so per­haps we should talk about “mod­ify­ing our minds” and not our prefs, since it’s hard to com­pletely ex­clude most of our mind from the pro­cess.

• Then it’s easy (well, eas­ier) to rea­son about chang­ing their prefer­ences be­cause we can hold the other parts con­stant.

As per Pei Wang’s sug­ges­tion, I’m stat­ing that I’m go­ing to opt out of this con­ver­sa­tion un­til you take se­ri­ously (ac­cept/​in­ves­ti­gate/​ar­gue against) the state­ment that prefer­ence is not to be mod­ified, some­thing that I stressed in sev­eral of the last com­ments.

• There are other rele­vant differ­ences as well, of course. For in­stance, a good ra­tio­nal agent would be able to liter­ally rewrite its prefer­ences, while hu­mans have trou­ble with self-bind­ing their fu­ture selves.

• All pos­si­ble wor­lds are real, and prob­a­bil­ities rep­re­sent how much I care about each world.

Could you elab­o­rate on what it means to have a given amount of “care” about a world? For ex­am­ple, sup­pose that I as­sign (or ought to as­sign) prob­a­bil­ity 0.5 to a coin’s com­ing up heads. How do you trans­late this prob­a­bil­ity as­sign­ment into lan­guage in­volv­ing amounts of care for wor­lds?

• You care equally for your selves that see heads and your selves that see tails. If you don’t care what hap­pens to you af­ter you see heads, then you would as­sign prob­a­bil­ity one to tails. Of course, you’d be wrong in about half the wor­lds, but hey, no skin off your nose. You’re the one who sees tails. Those other guys … they don’t mat­ter.

• A bizarre in­ter­pre­ta­tion.

For ex­am­ple, car­ing about “liv­ing un­til to­mor­row” does not nor­mally mean as­sign­ing a zero prob­a­bil­ity to death in the in­terim. If any­thing that would tend to make you fear­less—in­differ­ent to whether you stepped in front of a bus or not—the very op­po­site of what we nor­mally mean by “car­ing” about some out­come.

• Thanks. That makes it a lot clearer.

It seems like this “car­ing” could be an­a­lyzed a lot more, though. For ex­am­ple, sup­pose I were an al­tru­ist who con­tinued to care about the “heads” wor­lds even af­ter I learned that I’m not in them. Wouldn’t I still as­sign prob­a­bil­ity ~1 to the propo­si­tion that the coin came up tails in my own world? What does that prob­a­bil­ity as­sign­ment of ~1 mean in that case?

I sup­pose the idea is that a prob­a­bil­ity cap­tures not only how much I care about a world, but also how much I think that I can in­fluence that world by act­ing on my val­ues.