# Formalizing Newcomb’s

This post was in­spired by taw urg­ing us to math­e­ma­tize New­comb’s prob­lem and Eliezer tel­ling me to post stuff I like in­stead of com­plain­ing.

To make New­comb’s prob­lem more con­crete we need a work­able model of Omega. Let me count the ways:

1) Omega reads your de­ci­sion from the fu­ture us­ing a time loop. In this case the con­tents of the boxes are di­rectly causally de­ter­mined by your ac­tions via the loop, and it’s log­i­cal to one-box.

2) Omega simu­lates your de­ci­sion al­gorithm. In this case the de­ci­sion al­gorithm has in­dex­i­cal un­cer­tainty on whether it’s be­ing run in­side Omega or in the real world, and it’s log­i­cal to one-box thus mak­ing Omega give the “real you” the mil­lion.

3) Omega “scans your brain and pre­dicts your de­ci­sion” with­out simu­lat­ing you: calcu­lates the FFT of your brain­waves or what­ever. In this case you can in­tend to build an iden­ti­cal scan­ner, use it on your­self to de­ter­mine what Omega pre­dicted, and then do what you please. Hilar­ity en­sues.

(NB: if Omega pro­hibits agents from us­ing me­chan­i­cal aids for self-in­tro­spec­tion, this is in effect a re­stric­tion on how ra­tio­nal you’re al­lowed to be. If so, all bets are off—this wasn’t the deal.)

(Another NB: this case is dis­tinct from 2 be­cause it re­quires Omega, and thus your own scan­ner too, to ter­mi­nate with­out simu­lat­ing ev­ery­thing. A simu­la­tor Omega would go into in­finite re­cur­sion if treated like this.)

4) Same as 3, but the uni­verse only has room for one Omega, e.g. the God Almighty. Then ipso facto it can­not ever be mod­el­led math­e­mat­i­cally, and let’s talk no more.

I guess this one is set­tled, folks. Any ques­tions?

• Well… for what­ever it’s worth, the case I as­sume is (3).

“Rice’s The­o­rem” pro­hibits Omega from do­ing this with all pos­si­ble com­pu­ta­tions, but not with hu­mans. It’s prob­a­bly not even all that difficult: peo­ple seem strongly at­tached to their opinions about New­comb’s Prob­lem, so their ac­tual move might not be too difficult to pre­dict. Any mind that has an un­der­stand­able rea­son for the move it fi­nally makes, is not all that difficult to simu­late at a high-level; you are do­ing it ev­ery time you imag­ine what it would do!

Omega is as­sumed to be in a su­pe­rior po­si­tion, but doesn’t re­ally need to be. I mean, I have no trou­ble imag­in­ing Omega as de­scribed—Omega figures out the de­ci­sion I come to, then acts ac­cord­ingly. Un­til I ac­tu­ally come to a de­ci­sion, I don’t know what Omega has already done—but of course my de­ci­sion is sim­ple: I take only box B. End of sce­nario.

If you’re try­ing to figure out what Omega will do first—well, you’re just do­ing that so that you can take both boxes, right? You just want to figure out what Omega does “first”, and then take both boxes any­way. So Omega knows that, re­gard­less of how much you in­sist that you want to com­pute Omega “first”, and Omega leaves box B empty. You re­al­ize this and take both boxes. End of sce­nario again.

You may have some odd ideas left about free will. Omega can not only pre­dict you, but prob­a­bly do it with­out much trou­ble. Some hu­mans might be able to take a pretty good guess too. Re: free will, see rele­vant posts, e.g. this.

But this is an an­cient dilemma in de­ci­sion the­ory (much like free will in philos­o­phy), of which one should Google “causal de­ci­sion the­ory”, “ev­i­den­tial de­ci­sion the­ory”, and “New­comblike” for en­light­en­ment.

• My strat­egy. I build a ma­chine learn­ing pro­gram that takes in half the data available about Omega and how well it pre­dicts peo­ple who are likely to perform com­plex strate­gies, and data mines on that. If the com­puter pro­gram man­ages a high ac­cu­racy on the pre­dict­ing the test set, and shows a sig­nifi­cant chance that it will pre­dict me to one box, then I two box.

Other­wise I one box.

Rea­son­ing, it should be fairly ob­vi­ous from this strat­egy that I am likely to one box, pre­dict­ing Omega be­ing hard. So if I can tell Omega is likely to pre­dict this and I can pre­dict Omega ac­cu­rately, I’ll then two box.

The goal is to try to force Omega into pre­dict­ing that I will one box, while be­ing more pow­er­ful than Omega in pre­dic­tive power.

Not sure this will work, I’d like to try to do the math at some point.

• My strat­egy. I build a ma­chine learn­ing pro­gram that takes in half the data available about Omega and how well it pre­dicts peo­ple who are likely to perform com­plex strate­gies, and data mines on that. If the com­puter pro­gram man­ages a high ac­cu­racy on the pre­dict­ing the test set, and shows a sig­nifi­cant chance that it will pre­dict me to one box, then I two box. … The goal is to try to force Omega into pre­dict­ing that I will one box, while be­ing more pow­er­ful than Omega in pre­dic­tive power.

Dunno, you’d have to pay me a lot more than \$1000 to go to all that trou­ble. Doesn’t seem ra­tio­nal to do all that work just to get an ex­tra \$1000 and a tem­po­rary feel­ing of su­pe­ri­or­ity.

• I dunno. I think I could make a ‘ma­chine learn­ing pro­gram’ that can pre­dict a test set of ‘ev­ery guess out of 1,000,000 was right’ pretty quickly.

• Aren’t these rather duck­ing the point? The situ­a­tions all seem to be as­sum­ing that we our­selves have Omega-level in­for­ma­tion and re­sources, in which case why do we care about the money any­way? I’d say the rele­vant cases are:

3b) Omega uses a scan­ner, but we don’t know how the scan­ner works (or we’d be Omega-level en­tities our­selves).

5) Omega is us­ing one of the above meth­ods, or one we haven’t thought of, but we don’t know which. For all we know he could be read­ing the an­swers we gave on this blog post, and is just re­ally good at guess­ing who will stick by what they say, and who won’t. Un­less we ac­tu­ally know the method with suffi­cient con­fi­dence to risk los­ing the mil­lion, we should one-box. (: Origi­nally wrote two-box here—I meant to say one-box)

• 3b) Our ig­no­rance doesn’t change the fact that, if the scan­ner is in prin­ci­ple re­peat­able, re­al­ity con­tains a con­tra­dic­tion. Type 3 is just im­pos­si­ble.

5) If I were in this situ­a­tion, I’d as­sume a prior over pos­si­ble Omegas that gave large weight to types 1 and 2, which means I would one-box. My prior is jus­tified be­cause a work­able Omega of type 3 or 4 is harder for me to imag­ine than 1 or 2. Disagree? What would you do as a good Bayesian?

• Type 3 is just im­pos­si­ble.

No—it just means it can’t be perfect. A scan­ner that works 99.9999999% of the time is effec­tively in­dis­t­in­guish­able from a 100% for the pur­pose of the prob­lem. One that is 100% ex­cept in the pres­ence of re­cur­sion is com­pletely iden­ti­cal if we can’t con­struct such a scan­ner.

My prior is jus­tified be­cause a work­able Omega of type 3 or 4 is harder for me to imag­ine than 1 or 2. Disagree? What would you do as a good Bayesian?

I would one-box, but I’d do so re­gard­less of the method be­ing used, un­less I was con­fi­dent I could bluff Omega (which would gen­er­ally re­quire Omega-level re­sources on my part). It’s just that I don’t think the ex­act im­ple­men­ta­tion Omega uses (or even whether we know the method) ac­tu­ally mat­ter.

• This is a good post. It ex­plains that “given any con­crete im­ple­men­ta­tion of Omega, the para­dox ut­terly dis­ap­pears.”

• I’m quite both­ered by Eliezer’s lack of in­put to this thread. To me this seems like the most valuable thread of New­comb’s we had on OB/​LW, and he’s the biggest fan of the prob­lem here, so I would have guessed he thought about it a lot, and tried some mod­els even if they failed. Yet he didn’t write any­thing here. Why is it so?

• Be­cause the dis­cus­sion here didn’t seem in­ter­est­ing rel­a­tive to the dis­cus­sions I’ve already read in philos­o­phy; see the ed­ited vol­ume Para­doxes of Ra­tion­al­ity and Co­op­er­a­tion or start googling on “ev­i­den­tial de­ci­sion the­ory” and “causal de­ci­sion the­ory”.

I’ve never launched into a full-fledged dis­cus­sion of New­comb’s Prob­lem be­cause that would quickly de­gen­er­ate into a full-blown se­quence in which I pre­sented the gen­eral solu­tion (ten­ta­tively la­beled “time­less de­ci­sion the­ory”).

From my per­spec­tive this is a big, difficult, com­pli­cated, long-stand­ing, con­tro­ver­sial, overde­ter­mined, el­e­gant, solved prob­lem, like the in­ter­pre­ta­tion of quan­tum me­chan­ics. Though in both cases there’s a cou­ple of lef­tover prob­lems, the Born statis­tics for QM and some mat­ters of math­e­mat­i­cal rep­re­sen­ta­tion for New­comb, which may or may not rep­re­sent a gate­way to other mys­ter­ies af­ter the origi­nal main prob­lem has been solved.

I’ll re­peat yet again my stand­ing offer to do my PhD the­sis on New­comblike prob­lems if any­one will let me come in and just do a PhD the­sis rather than de­mand­ing 8 years of class at­ten­dance.

• Eliezer,

If what you have is good enough for a PhD the­sis, you should just pub­lish the thing as a book and then ap­ply for a PhD based on prior work. On the other hand, there are plenty of schools with pure re­search de­grees that will let you write a PhD with­out course­work (mostly in UK) but they won’t likely let you in with­out a de­gree or some re­ally im­pres­sive al­ter­na­tive cre­den­tials. But then, you prob­a­bly have the lat­ter.

• All uni­ver­si­ties that I know of only grant PhDs based on prior work to their own pre­vi­ous stu­dents who’ve already taken a Masters there. If there is any uni­ver­sity that just grants PhDs for suffi­ciently good prior work, do let me know.

• For a cer­tain defi­ni­tion of suffi­ciently good prior work, uni­ver­si­ties will grant PhDs. When I was in high school, I took a sum­mer pro­gram at CMU and the pro­fes­sor Steven Ru­dich said that if we were to prove P=NP or P!=NP or prove it un­de­cid­able or what­ever, that would be good for an in­stant PhD from CMU. I’m pretty sure the prob­lem he referred to was P/​NP, but it’s been a while and it may have been an­other Millen­nium Prob­lem.

So if you hap­pen to have a proof for P/​NP sit­ting around, let me know and I’ll in­tro­duce you to Dr. Ru­dich.

• In­deed. I’d thought De Mont­fort offered a PhD based on prior work, but can’t seem to find a refer­ence for it. I’ve also heard that the Univer­sity of Lu­ton (which would now be the Univer­sity of Bed­ford­shire) would do them. How­ever in ei­ther case, you’d likely need at least a bach­e­lor’s de­gree, so that seems like a dead end.

But maybe you can do some­thing re­ally im­pres­sive and get one of those ‘hon­orary’ doc­torates. I hear they’re as good as real ones.

• Pre­sum­ably the last line is sar­casm, but it’s hard to tell over the In­ter­net.

• No, I was be­ing se­ri­ous. I’m pretty sure if you, say, do some­thing No­bel Prize-wor­thy, some­one will hop to and give you an hon­orary doc­torate, and no­body will deny you’ve earned it.

• Honorary doc­torates are rou­tinely handed out to ran­dom for­eign dig­ni­taries or peo­ple who donate money to col­leges, and do not en­ti­tle the bearer to be called “Dr.”

Kurzweil has 16 hon­orary doc­torates plus the Na­tional Medal of Tech­nol­ogy and he still gets writ­ten up as “Mr. Kurzweil”.

• Honorary doc­torates are rou­tinely handed out to ran­dom for­eign dig­ni­taries or peo­ple who donate money to col­leges, and do not en­ti­tle the bearer to be called “Dr.”

I wish. I’m think­ing of a friend’s boss, a pri­vate school head­mas­ter, who in­sists on wav­ing around his hon­orary doc­torate as “Dr. [name]”. The friend, who was teach­ing there, has an ac­tual proper sweat of the brain Ph.D, and he in­sisted she should be ad­dressed as “Mrs. [name]”. WHAT.

• Good point. At any rate, I’ll keep an eye out for any doc­torates by prior work from ac­cred­ited schools and drop you a line.

• thom: you’re just wast­ing time sug­gest­ing this. It’s been brought up on SL4 mul­ti­ple times, and the peo­ple ar­gu­ing like you have been in­effec­tive each time.

• I’d ap­pre­ci­ate a short ex­tended ab­stract of what you’ve col­lected (on re­lated tech­ni­cal top­ics), with­out ex­pla­na­tions, just out­lin­ing what it’s about and link­ing to the key­words. I’m cur­rently go­ing through the stage of for­mal­iz­ing the ear­lier in­tu­itions, and it looks like a huge syn­the­sis, lots of stuff yet to learn, so some fo­cus may be use­ful.

• Sorry, too huge. There’s a nice dis­ser­ta­tion on the sub­ject here: http://​​kops.ub.uni-kon­stanz.de/​​vol­l­texte/​​2000/​​524/​​pdf/​​led­wig.pdf

• I think I grasp this prob­lem well enough, I’m not sure it’s use­ful to plough through the ex­ist­ing philos­o­phy at this point (am I wrong, is there some­thing tech­ni­cally use­ful in e.g. that the­sis?).

The ex­am­ples of prob­lems I was try­ing to figure out these last weeks is e.g. rep­re­sen­ta­tion of prefer­ence or­der (lat­tices vs. prob­a­bil­ities vs. graph­i­cal mod­els vs. other math­e­mat­i­cal struc­tures), re­la­tion and con­ver­sions be­tween differ­ent rep­re­sen­ta­tions of the state space (vari­ables/​pred­i­cates/​etc.), rep­re­sen­ta­tion of one agent by an­other, “agents” as effi­cient ab­strac­tions of reg­u­lar­i­ties in the prefer­ence or­der, com­pound prefer­ences and more global op­ti­miza­tion re­sult­ing from co­op­er­a­tion of mul­ti­ple agents, in­clud­ing the coun­ter­fac­tual agents and agents act­ing at differ­ent lo­cal ar­eas in time/​space/​rep­re­sen­ta­tion of state space, etc.

• rep­re­sen­ta­tion of prefer­ence or­der (lat­tices vs. prob­a­bil­ities vs. graph­i­cal mod­els vs. other math­e­mat­i­cal struc­tures), re­la­tion and con­ver­sions be­tween differ­ent rep­re­sen­ta­tions of the state space (vari­ables/​pred­i­cates/​etc.)

There’s ac­tu­ally quite a lot of this in James Joyce’s The foun­da­tions of causal de­ci­sion the­ory, at what ap­pears to me to be a gra­tu­itiously high math level.

• (5) Omega uses or­di­nary con­jur­ing, or heretofore-un­known pow­ers to put the mil­lion in the box af­ter you make your de­ci­sion. Solu­tion: one-box for sure, no de­ci­sion the­ory trick­ery needed. This would be in prac­tice the con­clu­sion we would come to if we en­coun­tered a be­ing that ap­peared to be­have like Omega, and there­fore is also the an­swer in any sce­nario where we don’t know the true im­ple­men­ta­tion of Omega (ie any real sce­nario).

If the boxes are trans­par­ent, re­solve to one-box iff the big box is empty.

• Good! Now we have some ter­minol­ogy for fu­ture gen­er­a­tions:

1) Tem­po­ral Omega 2) Si­mu­la­tor Omega 3) Ter­mi­nat­ing Omega 4) Sin­gle­ton Omega 5) Cheat­ing Omega

Great point about the prior, thanks.

• I out­lined a few more pos­si­bil­ities on Over­com­ing Bias last year:

There are many ways Omega could be do­ing the pre­dic­tion/​place­ment and it may well mat­ter ex­actly how the prob­lem is set up. For ex­am­ple, you might be de­ter­minis­tic and he is pre­calcu­lat­ing your choice (much like we might be able to do with an in­sect or com­puter pro­gram), or he might be us­ing a quan­tum suicide method, (quan­tum) ran­dom­iz­ing whether the mil­lion goes in and then de­stroy­ing the world iff you pick the wrong op­tion (This will lead to us ob­serv­ing him be­ing cor­rect 100100 times as­sum­ing a many wor­lds in­ter­pre­ta­tion of QM). Or he could have just got lucky with the last 100 peo­ple he tried it on.

If it is the de­ter­minis­tic op­tion, then what do the coun­ter­fac­tu­als about choos­ing the other box even mean? My ap­proach is to say that ‘You could choose X’ means that if you had de­sired to choose X, then you would have. This is a stan­dard way of un­der­stand­ing ‘could’ in a de­ter­minis­tic uni­verse. Then the an­swer de­pends on how we sup­pose the world to be differ­ent to give you coun­ter­fac­tual de­sires. If we do it with a mir­a­cle near the mo­ment of choice (his­tory is the same, but then your de­sires change non-phys­i­cally), then you ought two-box as Omega can’t have pre­dicted this. If we do it with an ear­lier mir­a­cle, or with a change to the ini­tial con­di­tions of the uni­verse (the Tannsjo in­ter­pre­ta­tion of coun­ter­fac­tu­als) then you ought one-box as Omega would have pre­dicted your choice. Thus, if we are un­der­stand­ing Omega as ex­trap­o­lat­ing your de­ter­minis­tic think­ing, then the an­swer will de­pend on how we un­der­stand the coun­ter­fac­tu­als. One-box­ers and Two-box­ers would be peo­ple who in­ter­pret the nat­u­ral coun­ter­fac­tual in the ex­am­ple in differ­ent (and equally valid) ways.

If we un­der­stand it as Omega us­ing a quan­tum suicide method, then the ob­jec­tively right choice de­pends on his ini­tial prob­a­bil­ities of putting the mil­lion in the box. If he does it with a 50% chance, then take just one box. There is a 50% chance the world will end ei­ther choice, but this way, in the case where it doesn’t, you will have a mil­lion rather than a thou­sand. If, how­ever, he uses a 99% chance of putting noth­ing in the box, then one-box­ing has a 99% chance of de­stroy­ing the world which dom­i­nates the value of the ex­tra money, so in­stead two-box, take the thou­sand and live.

If he just got lucky a hun­dred times, then you are best off two-box­ing.

If he time trav­els, then it de­pends on the na­ture of time-travel...

Thus the an­swer de­pends on key de­tails not told to us at the out­set. Some peo­ple ac­cuse all philo­soph­i­cal ex­am­ples (like the trol­ley prob­lems) of not giv­ing enough in­for­ma­tion, but in those cases it is fairly ob­vi­ous how we are ex­pected to fill in the de­tails. This is not true here. I don’t think the New­comb prob­lem has a sin­gle cor­rect an­swer. The value of it is to show us the differ­ent pos­si­bil­ities that could lead to the situ­a­tion as speci­fied and to see how they give differ­ent an­swers, hope­fully illu­mi­nat­ing the topic of free-will, coun­ter­fac­tu­als and pre­dic­tion.

• There’s a (6) which you might con­sider a var­i­ant of (5): hav­ing made his best guess on whether you’re go­ing to go­ing to one-box or two-box, Omega en­forces that guess with or­bital mind con­trol lasers.

• That’s a cre­ative at­tempt to avoid re­ally con­sid­er­ing New­comb’s prob­lem; but as I sug­gested ear­lier, the noisy real-world ap­pli­ca­tions are real enough to make this a ques­tion worth con­fronting on its own terms.

Least Con­ve­nient Pos­si­ble World: Omega is type (3), and does not offer the game at all if it calcu­lates that its an­swers turn out to be con­tra­dic­tions (as in your ex­am­ple above). At any rate, you’re not ca­pa­ble of build­ing or ob­tain­ing an ac­cu­rate Omega’ for your pri­vate use.

Aside: If Omega sees prob­a­bil­ity p that you one-box, it puts the mil­lion dol­lars in with prob­a­bil­ity p, and in ei­ther case writes p on a slip of pa­per in that box. Omega has been shown to be ex­tremely well-cal­ibrated, and its p only differs sub­stan­tially from 0 or 1 in the case of the jok­ers who’ve tried us­ing a ran­dom pro­cess to out­wit it. (I always thought this would be an el­e­gant solu­tion to that prob­lem; and note that the ex­pected value of 1-box­ing with prob­a­bil­ity p should then be 1000000p+1000(1-p).)

Yes, these are ex­tra rules of the game. But if these re­stric­tions make ra­tio­nal­ity im­pos­si­ble, then it doesn’t seem hu­man be­ings can be ra­tio­nal by your stan­dards (as we’re already be­ing mod­eled fairly of­ten in so­cial life)— in which case, we’ll take what­ever Art is our best hope in­stead, and call that ra­tio­nal­ity.

So what do you do in this situ­a­tion?

• Eliezer has re­peat­edly stated in dis­cus­sions of NP that Omega only cares about the out­come, not any par­tic­u­lar “rit­ual of cog­ni­tion”. This is an es­sen­tial part of the puz­zle be­cause once you start pun­ish­ing agents for their rea­son­ing you might as well go all the way: re­ward only ir­ra­tional agents and say nyah nyah puny ra­tio­nal­ists. Your Omega bounds how ra­tio­nal I can be and out­right for­bids think­ing cer­tain thoughts. In other words, the origi­nal rai­son d’etre was re­fin­ing the no­tion of perfect ra­tio­nal­ity, whereas your for­mu­la­tion is about ap­prox­i­ma­tions to ra­tio­nal­ity. Well, who defines what is a good ap­prox­i­ma­tion and what isn’t? I’m gonna one-box with­out ex­pla­na­tion and call this ra­tio­nal­ity. Is this bad? By what met­ric?

Believe or not, I have con­sid­ered the most in­con­ve­nient wor­lds re­peat­edly while writ­ing this, or I would have had just one or two cases in­stead of four.

• A strat­egy Omega uses to avoid para­dox which has the effect of pun­ish­ing cer­tain rit­u­als of cog­ni­tion be­cause they lead to para­dox is differ­ent than Omega de­liber­ately hand­i­cap­ping your thought pro­cess. It is not a win­ning strat­egy to pur­sue a line of thought that pro­duces a para­dox in­stead of a win­ning de­ci­sion. I would wait un­til Omega for­bids strate­gies that would oth­er­wise win be­fore com­plain­ing that he “bounds how ra­tio­nal I can be”.

• Maybe see it as a com­pe­ti­tion of wits. Between two agents whose per­sonal goal is or isn’t com­pat­i­ble. If they are not of similar ca­pa­bil­ity, the one with more com­pu­ta­tional re­sources, and how well those re­sources are be­ing used, is the one which will get its way, against the other’s will if nec­es­sary. If you were “big­ger” than omega, then you’d be the one to win, no mat­ter which weird rules omega would wish to use. But omega is big­ger … by defi­ni­tion.

In this case, the only way for the smaller agent to suc­ceeds is to em­bed his own goals into the other agent’s. In prac­tice agents aren’t om­ni­scient or om­nipo­tent, so even an agent or­ders of mag­ni­tude more pow­er­ful than an­other, may still fail against the lat­ter. That would be­come in­creas­ingly un­likely, but not to­tally im­pos­si­ble (as in, play­ing lot­ter­ies).

If the differ­ence in power is even small enough, then both agents ought to co­op­er­ate and com­pro­mise, both, since in most cases that’s how they can max­i­mize their gains.

But in the end, once again, ra­tio­nal­ity is about re­li­ably win­ning in as many cases as pos­si­ble. In some cases, how­ever un­likely and un­nat­u­ral they may seem, it just can’t be achieved. That’s what op­ti­miza­tion pro­cesses, and how pow­er­ful they are, are about. They steer the uni­verse into very un­likely states. In­clud­ing states where “ra­tio­nal­ity” is coun­ter­pro­duc­tive.

• Maybe see it as a com­pe­ti­tion of wits.

Yes! Where is the money? A bat­tle of wits has be­gun! It ends when a box is opened.

Of course, it’s so sim­ple. All I have to do is di­v­ine from what I know of Omega: is it the sort of agent who would put the money in one box, or both? Now, a clever agent would put lit­tle money into only one box, be­cause it would know that only a great fool would not reach for both. I am not a great fool, so I can clearly not take only one box. But Omega must have known I was not a great fool, and would have counted on it, so I can clearly not choose both boxes.

Truly, Omega must ad­mit that I have a dizzy­ing in­tel­lect.

On the other hand, per­haps I have con­fused this with some­thing else.

• My ver­sion of Omega still only cares about its pre­dic­tion of your de­ci­sion; it just so hap­pens that it doesn’t offer the game if it pre­dicts “you will 2-box if and only if I pre­dict you will 1-box”, and it plays prob­a­bil­is­ti­cally when it pre­dicts you de­cide prob­a­bil­is­ti­cally. It doesn’t re­ward you for your de­ci­sion al­gorithm, only for its out­come— even in the above cases.

Yes, I agree this is about ap­prox­i­ma­tions to ra­tio­nal­ity, just like Bayescraft is about ap­prox­i­mat­ing the ideal of Bayesian up­dat­ing (im­pos­si­ble for us to achieve since com­pu­ta­tion is costly, among other things). I tend to think such ap­prox­i­ma­tions should be ro­bust even as our limi­ta­tions diminish, but that’s not some­thing I’m con­fi­dent in.

Well, who defines what is a good ap­prox­i­ma­tion and what isn’t?

A cluster in con­ceptspace. Bet­ter ap­prox­i­ma­tions should have more, not less, ac­cu­rate maps of the ter­ri­tory and should steer higher pro­por­tions of the fu­ture into more de­sir­able re­gions (with re­spect to our prefer­ences).

I’m gonna one-box with­out ex­pla­na­tion and call this ra­tio­nal­ity. Is this bad? By what met­ric?

I think “with­out ex­pla­na­tion” is bad in that it fails to gen­er­al­ize to similar situ­a­tions, which I think is the whole point. In deal­ing with agents who model your own de­ci­sions in ad­vance, it’s good to have a gen­eral the­ory of ac­tion that sys­tem­at­i­cally wins against other the­o­ries.

• Your fix is a kludge. I could ran­dom­ize: use the de­tec­tor to de­ter­mine Omega’s p and then use 1-p, or some­thing like that. Give me a gen­eral de­scrip­tion of what your Omega does, and I’ll give you a con­tra­dic­tion in the spirit of my origi­nal post. Patch the holes all you want. Pre­dict­ing the fu­ture always in­volves a con­tra­dic­tion, it’s just more or less hard to tease out. You can’t pre­dict the fu­ture and out­law con­tra­dic­tions by fiat; it is log­i­cally im­pos­si­ble. This was one of the points of my post.

• Your fix is a bit of a kludge. I could ran­dom­ize: use my de­tec­tor to de­ter­mine p, and then use 1-p. So for to­tal con­sis­tency you should amend Omega to “pro­tect” the value of p, and ban the agent if p is tam­pered with. Now it sounds bul­let­proof, right?

But here’s the rub: the agent doesn’t need a perfect replica of Omega. A half-assed one will do fine. In fact, if a cer­tain method of in­tro­spec­tion into your ini­tial state al­lowed Omega to de­ter­mine the value of p, then any weak at­tempt at in­tro­spec­tion will give you some small but non-zero in­for­ma­tion about what p Omega de­tected. So ev­ery liv­ing per­son will fail your Omega’s test. My idea with the scan­ner was just a way to “ex­ter­nal­ize” the in­tro­spec­tion, mak­ing the con­tra­dic­tion stark and ev­i­dent.

Any other ideas on how Omega should be­have?

• I could ran­dom­ize: use my de­tec­tor to de­ter­mine p, and then use 1-p.

In this case, Omega figures out you would use that de­tec­tor and pre­dicts you will use 1-p. If your de­tec­tor is effec­tive, it will take into ac­count that Omega knows about it, and will figure that Omega pre­dicted 1-(1-p) = p. But Omega would have re­al­ized that the de­tec­tor could do that. This is the be­gin­ning of an in­finite re­cur­sion at­tempt­ing to re­solve a para­dox, no differ­ent be­cause we are us­ing prob­a­bil­ities in­stead of Booleans. Omega rec­og­nizes this and con­cludes the game is not worth play­ing. If you and your de­tec­tor are ra­tio­nal, you should too, and find a differ­ent strat­egy. (Well, Omega could pre­dict a prob­a­bil­ity of .5 which is sta­ble, but a strat­egy to take ad­van­tage of this would lead to para­dox.)

• Omegas of type 3 don’t use simu­la­tions. If Omega is a simu­la­tor, see case 2.

...why is ev­ery­body latch­ing on to 3? A brain­wave-read­ing Omega is a pa­thetic joke that took no effort to kill. Any re­al­is­tic Omega would have to be type 2 any­way.

Para­doxes show that your model is bad. My post was about defin­ing non-con­tra­dic­tory mod­els of New­comb’s prob­lem and see­ing what we can do with them.

• Could you taboo “simu­la­tion” and ex­plain what you are pro­hibit­ing Omega from do­ing by spec­i­fy­ing that Omega does not use simu­la­tions? Pre­sum­ably this still al­lows Omega to make pre­dic­tions.

• That one’s sim­ple: pro­hibit in­dex­i­cal un­cer­tainty. I must be able to as­sume that I am in the real world, not in­side Omega. So should my scan­ner’s in­ter­nal com­pu­ta­tion—if I an­ti­ci­pate it will be run in­side Omega, I will change it ac­cord­ingly.

Edit: sorry, now I see why ex­actly you’re asked. No, I have no proof that my list of Omega types is ex­haus­tive. There could be a mid­dle ground be­tween types 2 and 3: an Omega that doesn’t simu­late you, but still some­how pro­hibits you from us­ing an­other Omega to cheat. But, as or­thonor­mal’s ex­am­ples show, such a ma­chine doesn’t read­ily spring to mind.

• In­dex­i­cal un­cer­tainty is a prop­erty of you, not Omega.

Say­ing Omega can­not cre­ate a situ­a­tion in which you have in­dex­i­cal un­cer­tainty is too vague. What pro­cess of cog­ni­tion is pro­hibited to Omega that pre­vents pro­duc­ing in­dex­i­cal un­cer­tainty, but still al­lows for mak­ing cal­ibrated, dis­crim­i­nat­ing pre­dic­tions?

• You’re dig­ging deep. I already ad­mit­ted that my list of Omegas isn’t proven to be ex­haus­tive and prob­a­bly can never be, given how crazy the in­di­vi­d­ual cases sound. The thing I call a type 3 Omega should bet­ter be called a Ter­mi­nat­ing Omega, a de­vice that out­puts one bit in bounded time given any in­put situ­a­tion. If Omega is non-ter­mi­nat­ing—e.g. it throws me out of the game on pre­dict­ing cer­tain be­hav­ior, or hangs for­ever on some in­puts—of course such an Omega doesn’t nec­es­sar­ily have to be a simu­la­tion. But then you need a halfway cred­ible ac­count of what it does, be­cause oth­er­wise the prob­lem is un­for­mu­lated and in­com­plete.

The pro­cess you’ve de­scribed (Omega re­al­izes this, then re­al­izes that...) sounded like a simu­la­tion—that’s why I referred you to case 2. Of course you might have meant some­thing I hadn’t an­ti­ci­pated.

• Part of my mo­ti­va­tion for dig­ging deep on this is­sue is that, al­though I did not in­tend for my de­scrip­tion of Omega and the de­tec­tor rea­son­ing about each other to be based on a simu­la­tion, I could see af­ter you brought it up that it might be in­ter­preted that way. I thought if I knew on a more de­tailed level what we mean by “simu­la­tion”, I would be able to tell if I had im­plic­itly as­sumed that Omega was us­ing one. How­ever, any strat­egy I come up with for mak­ing pre­dic­tions seems like some­thing I could con­sider a simu­la­tion, though it might lack de­tail, and through omit­ting im­por­tant de­tails, be in­ac­cu­rate. Even just guess­ing could be con­sid­ered a very un­de­tailed, very in­ac­cu­rate simu­la­tion.

I would like a defi­ni­tion of simu­la­tion that doesn’t lead to this con­clu­sion, but in case there isn’t one, sup­pose the re­stric­tion against simu­la­tion re­ally means that Omega does not use a perfect simu­la­tion, and you have a chance to re­solve the in­dex­i­cal un­cer­tainty.

I can imag­ine situ­a­tions in which an in­com­plete, though still highly ac­cu­rate, simu­la­tion pro­vides in­for­ma­tion to the simu­lated sub­ject to re­solve the in­dex­i­cal un­cer­tainty, but this in­for­ma­tion is difficult or even im­pos­si­ble to in­ter­pret.

For ex­am­ple, sup­pose Omega does use a perfect simu­la­tion, ex­cept that he flips a coin. In the real world, Omega shows you the true re­sult of the coin toss, but he simu­lates your re­sponse as if he shows you the op­po­site re­sult. Now you still don’t know if you are in a simu­la­tion or re­al­ity, but you are no longer guaran­teed by de­ter­minism to make the same de­ci­sion in each case. You could one box if you see heads and two box if you see tails. If you did this, you have a 50% prob­a­bil­ity that the true flip was heads, so you gain noth­ing, and a 50% prob­a­bil­ity that the true flip was tails and you gain \$1,001,000, for an ex­pected gain of \$500,500. This is not as good as if you just one box ei­ther way and gain \$1,000,000. If Omega in­stead flips a bi­ased coin that shows tails 60% of the time, and tells you this, then the same strat­egy has an ex­pected gain of \$600,600, still not as good as com­plete one-box­ing. But if the coin was bi­ased to show tails 1000 times out of 1001, then the strat­egy ex­pects to equal one-box­ing, and it will do bet­ter for a more ex­treme bias.

So, if you sup­pose that Omega uses an im­perfect simu­la­tion (with­out the coin), you can gather ev­i­dence about if you are in re­al­ity or the simu­la­tion. You would need to achieve a prob­a­bil­ity of greater than 1000/​1001 that you are in re­al­ity be­fore it is a good strat­egy to two box. I would be im­pressed with a strat­egy that could ac­com­plish that.

As for ter­mi­nat­ing, if Omega de­tects a para­dox, Omega puts money in box 1 with 50% prob­a­bil­ity. It is not a win­ning strat­egy to force this out­come.

• It seems your prob­a­bil­is­tic simu­la­tor Omega is amenable to ra­tio­nal anal­y­sis just like my case 2. In good im­ple­men­ta­tions we can’t cheat, in bad ones we can; it all sounds quite nor­mal and re­as­sur­ing, no trace of a para­dox. Just what I aimed for.

As for ter­mi­nat­ing, we need to de­mys­tify what it means by “de­tect­ing a para­dox”. Does it some­how com­pute the ac­tual prob­a­bil­ities of me choos­ing one or two boxes? Then what part of the world is as­sumed to be “ran­dom” and what part is eval­u­ated ex­actly? An an­swer to this ques­tion might clear things up.

• One way Omega might pre­vent para­dox is by adding an ar­bi­trary time limit, say one hour, for you to choose whether to one box or two box. Omega could then run the simu­la­tion, how­ever ac­cu­rate, up to the limit of simu­lated time, or when you ac­tu­ally make a de­ci­sion, whichever comes first. Ex­ceed­ing the time limit could be treated as iden­ti­cal to two box­ing. A more so­phis­ti­cated Omega that can search for a time in the simu­la­tion when you have made a de­ci­sion in con­stant time, per­haps by hav­ing the simu­la­tion state de­scribed by a closed form func­tion with nice alge­braic prop­er­ties, could sim­ply re­quire that you even­tu­ally make a de­ci­sion. This es­sen­tially puts the bur­den on the sub­ject not to cre­ate a para­dox, or any­thing that might be mis­taken for a para­dox, or just take too long to de­cide.

Then what part of the world is as­sumed to be “ran­dom” and what part is eval­u­ated ex­actly?

Well Omega could give you a pseudo ran­dom num­ber gen­er­a­tor, and agree to treat it as a prob­a­bil­is­tic black box when mak­ing pre­dic­tions. It might make sense to treat quan­tum de­co­her­ence as giv­ing prob­a­bil­ities to ob­serve the differ­ent macro­scopic out­comes, un­less some­thing like world man­gling is true and Omega can pre­dict de­ter­minis­ti­cally which wor­lds get man­gled. Less ac­cu­rate Omegas could use prob­a­bil­ity to ac­count for their own in­ac­cu­racy.

In good im­ple­men­ta­tions we can’t cheat, in bad ones we can

Even bet­ter, in prin­ci­pal, though it would be com­pu­ta­tion­ally difficult, de­scribe differ­ent simu­la­tions with differ­ent com­plex­ities and as­so­ci­ated Oc­cam pri­ors, and with differ­ent prob­a­bil­ities of Omega mak­ing cor­rect pre­dic­tions. From this we could de­ter­mine how much of a track record Omega needs be­fore we con­sider one box­ing a good strat­egy. Though I sus­pect ac­tu­ally do­ing this would be harder than mak­ing Omega’s pre­dic­tions.

• I find New­comb’s prob­lem in­ter­est­ing. Omega pre­dicts ac­cu­rately. This is im­pos­si­ble in my ex­pe­rience. We are not dis­cussing a prob­lem any of us is likely to face. How­ever I still find dis­cussing counter-fac­tu­als in­ter­est­ing.

To make New­comb’s prob­lem more con­crete we need a work­able model of Omega

I do not think that is the case. Whether Omega pre­dicts by time travel, mind-read­ing, or even re­moves money from the box by tele­por­ta­tion when it ob­serves the sub­ject tak­ing two boxes is a sep­a­rate dis­cus­sion, con­sid­er­ing laws of physics, SF, what­ever. This might be quite fun, but is wholly sep­a­rate from dis­cussing New­comb’s prob­lem it­self.

I think an abil­ity to dis­cuss a counter-fac­tual with­out hav­ing some way of re­lat­ing it to Real­ity is a use­ful skill. Play­ing around with the prob­lem, I think, has in­creased my un­der­stand­ing of the real World. Then the “need” to ex­plain how a real Omega might do what Omega is de­scribed as be­ing able to do just gets in the way.

• Play­ing around with the prob­lem, I think, has in­creased my un­der­stand­ing of the real World.

In what ways?

Most in­sights that arise from New­comb’s prob­lem seem to me to be ei­ther phony or deriv­able from sim­pler prob­lems that don’t fea­ture om­ni­scient en­tities. Ad­mit­tedly you can med­i­tate on the log­i­cal loop for­ever in the illu­sion that it in­creases your un­der­stand­ing. Maybe the un­ex­pected hang­ing para­dox will help snap you out? That para­dox also al­lows per­pet­ual med­i­ta­tion un­til we sit down and de­mys­tify the word “sur­prise” into math­e­mat­i­cal logic, ex­pos­ing the prob­lem state­ment as self-refer­en­tial and self-con­tra­dic­tory. In New­comb’s prob­lem we might just need to similarly de­mys­tify the word “pre­dict”, as I’ve been try­ing to.

• All right, I found an­other nice illus­tra­tion. Some philoso­phers to­day think that New­comb’s prob­lem is a model of cer­tain real-world situ­a­tions. Here’s a typ­i­cal spec­i­men of this idiocy, re­typed ver­ba­tim from here:

Let me de­scribe a typ­i­cal med­i­cal New­comb prob­lem. It has long been rec­og­nized that in peo­ple sus­cep­ti­ble to mi­graine, the on­set of an at­tack tends to fol­low the con­sump­tion of cer­tain foods, in­clud­ing choco­late and red wine. It has usu­ally been as­sumed that these foods are causal fac­tors, in some way trig­ger­ing at­tacks. This be­lief has been the source of much men­tal and phys­i­cal an­guish for those sus­cep­ti­ble both to mi­graines and to the at­trac­tions of these sub­stances. Re­cently how­ever an al­ter­na­tive the­ory has come to light. It has been dis­cov­ered that eat­ing choco­late is not a cause of mi­graine, but a joint effect of some pre-mi­grain­ous state (or ‘PMS’, as we doc­tors say). The phys­iolog­i­cal changes that com­prise PMS thus typ­i­cally in­crease a sub­ject’s de­sire for choco­late, as well as lead­ing, later, to the usual phys­i­cal symp­toms of mi­graine.

The ar­ti­cle goes on to sug­gest that, in a suffi­ciently freaky de­ci­sion the­ory, ab­stain­ing from choco­late can still help. Yes, folks, this is the best real-world sce­nario they could come up with. I rest my case .

• New­comb-like prob­lems arise when there is a causal thread pass­ing through your cog­ni­tive al­gorithm which pro­duces the cor­re­la­tion. There is no causal­ity go­ing through your cog­ni­tive al­gorithm to the mi­graine here. The au­thor doesn’t know what a new­comb-like prob­lem is.

• Some au­thors define “New­comblike prob­lem” as one that brings ev­i­den­tial and de­ci­sion the­ory into con­flict, which this does.

• So… in New­comb’s prob­lem, ev­i­den­tial says one-box, causal says two-box, causal clearly fails.

In Cho­co­late prob­lem, ev­i­den­tial says avoid choco­late, causal says eat the choco­late, ev­i­den­tial clearly fails.

Is that right?

• I as­sume it’s a typo: ev­i­den­tial vs. causal de­ci­sion the­o­ries.

Ev­i­den­tial de­ci­sion the­ory wins for the wrong rea­sons, and causal de­ci­sion the­ory just fails.

• But ev­i­den­tial ac­tu­ally tells you not to eat the choco­late? That’s a pretty spec­tac­u­lar failure mode—it seems like it could be ex­tended to not tak­ing your loved ones to the hos­pi­tal be­cause peo­ple tend to die there.

• Yeah, that was awk­wardly worded, I was only refer­ring to New­comb.

• I as­sume it’s a typo: ev­i­den­tial vs. causal de­ci­sion the­o­ries.

• In the stan­dard New­comb’s, is the deal Omega is mak­ing ex­plained to you be­fore Omega makes its de­ci­sion; and does the an­swer to my ques­tion mat­ter?

• Wikipe­dia says the deal is ex­plained be­fore­hand. It doesn’t seem to mat­ter in any of the mod­els pro­posed in the post and com­ments, but it could con­ceiv­ably mat­ter in some other model.

• NB: if Omega pro­hibits agents from us­ing me­chan­i­cal aids for self-in­tro­spec­tion, this is in effect a re­stric­tion on how ra­tio­nal you’re al­lowed to be. If so, all bets are off—this wasn’t the deal.

Sup­pose it was.

• Already an­swered above. If agents’ ra­tio­nal­ity is re­stricted, the prob­lem loses its origi­nal point of re­fin­ing “perfect ra­tio­nal­ity” and be­comes a ques­tion of ap­prox­i­ma­tions. Okay, my ap­prox­i­ma­tion: when con­fronted with a huge pow­er­ful agent that has a track record of 100% truth, be­lieve it. I one-box and win. Who are you to tell me my ap­prox­i­ma­tion is bad?

• Okay, my ap­prox­i­ma­tion: when con­fronted with a huge pow­er­ful agent that has a track record of 100% truth, be­lieve it. I one-box and win. Who are you to tell me my ap­prox­i­ma­tion is bad?

I don’t have prob­lems with that. But Omega doesn’t tell you “take one box to win”. It only tells that if you’ll take one box, it placed a mil­lion in it, and if you’ll take two boxes, it didn’t. It doesn’t tell which de­ci­sion you must take, the de­ci­sion is yours.

The whole thing is a test ground for de­ci­sion the­o­ries. If your de­ci­sion the­ory out­puts a de­ci­sion that you think is not the right one, then you need to work some more on that de­ci­sion the­ory, find­ing a way for it to com­pute the de­ci­sions you ap­prove of.

• An­noy­ance has it right but too cryp­tic: it’s the other way around. If your de­ci­sion the­ory fails on this test ground but works perfectly well in the real world, maybe you need to work some more on the test ground. For now it seems I’ve ad­e­quately demon­strated how your available op­tions de­pend on the im­ple­men­ta­tion of Omega, and look not at all like the de­ci­sion the­o­ries that we find effec­tive in re­al­ity. Good sign?

• An­noy­ance has it right but too cryp­tic: it’s the other way around. If your de­ci­sion the­ory fails on this test ground but works perfectly well in the real world, maybe you need to work some more on the test ground.

Not quite. The failure of a strong de­ci­sion the­ory on a test is a rea­son for you to start doubt­ing the ad­e­quacy of both the test prob­lem and the de­ci­sion the­ory. The de­ci­sion to amend one or the other must always come through you, un­less you already trust some­thing else more than you trust your­self. The para­dox doesn’t care what you do, it is merely a build­ing block to­wards bet­ter ex­pli­ca­tion of what kinds of de­ci­sions you con­sider cor­rect.

• Woah, let’s have some com­mon sense here in­stead of preach­ing. I have good rea­sons to trust ac­cepted de­ci­sion the­o­ries. What rea­son do I have to trust New­comb’s prob­lem? Given how much in my anal­y­sis turned out to de­pend on the im­ple­men­ta­tion of Omega, I don’t trust the thing at all any­more. Do you? Why?

• You are not asked to trust any­thing. You have a para­dox; re­solve it, un­der­stand it. What do you re­fer to, when us­ing the word “trust” above?

• Uh, didn’t I con­vince you that, given any con­crete im­ple­men­ta­tion of Omega, the para­dox ut­terly dis­ap­pears? Let’s go at it again. What kind of Omega do you offer me?

• The usual set­ting, you be­ing a suffi­ciently sim­ple mere hu­man, not build­ing your own Omegas in the pro­cess, go­ing through the pro­ce­dure in a con­trol­led en­vi­ron­ment if that helps to get the case stronger, and Omega be­ing able to pre­dict your ac­tual fi­nal de­ci­sion, by what­ever means it pleases. What the Omega does to pre­dict your de­ci­sion doesn’t af­fect you, shouldn’t con­cern you, it looks like only that it’s usu­ally right is rele­vant.

• “What the Omega does to pre­dict your de­ci­sion doesn’t af­fect you, shouldn’t con­cern you, it looks like only that it’s usu­ally right is rele­vant.”

Is this the least con­ve­nient world? What Omega does to pre­dict my de­ci­sion does con­cern me, be­cause it de­ter­mines whether I should one-box or two-box. How­ever, I’m will­ing to al­low that in a LCW, I’m not given enough in­for­ma­tion. Is this the New­comb “prob­lem”, then—how to make ra­tio­nal de­ci­sion when you’re not given enough in­for­ma­tion?

• No perfectly ra­tio­nal de­ci­sion the­ory can be ap­plied in this case, just like you can’t play chess perfectly ra­tio­nally with a desk­top PC. Sev­eral com­ments above I out­lined a good ap­prox­i­ma­tion that I would use and recom­mend a com­puter to use. This case is just… un­in­ter­est­ing. It doesn’t raise any ques­tion marks in my mind. It should?

• Can you please ex­plain why a ra­tio­nal de­ci­sion the­ory can­not be ap­plied?

• As I un­der­stand it, perfect ra­tio­nal­ity in this sce­nario re­quires we as­sume some Bayesian prior over all pos­si­ble im­ple­men­ta­tions of Omega and do a ton of com­pu­ta­tion for each case. For ex­am­ple, some Omegas could be type 3 and de­ceiv­able with non-zero prob­a­bil­ity; we have to de­ter­mine how. If we know which im­ple­men­ta­tion we’re up against, the calcu­la­tions are a lit­tle eas­ier, e.g. in the “simu­lat­ing Omega” case we just one-box with­out think­ing.

• By that defi­ni­tion of “perfect ra­tio­nal­ity” no two perfect ra­tio­nal­ists can ex­ist in the same uni­verse, or any ma­te­rial uni­verse in which the amount of elapsed time be­fore a de­ci­sion is always finite.

• Some as­sump­tions al­low you to play some games ra­tio­nally with finite re­sources, like in the last sen­tence of my pre­vi­ous com­ment. Un­for­tu­nately we aren’t given any such as­sump­tions in New­comb’s, so I fell back to the de­ci­sion pro­ce­dure recom­mended by you: Solomonoff in­duc­tion. Don’t like it? Give me a work­able model of Omega.

• Yes, it’s true. Perfectly play­ing any non-math­e­mat­i­cal “real world” game (the for­mu­la­tion Vladimir Nesov in­sists on) re­quires great pow­ers. If you can trans­late the game into maths to make it solv­able, please do.

• The de­ci­sion the­ory must al­low ap­prox­i­ma­tions, a rank­ing al­low­ing to find (rec­og­nize) as good a solu­tion as pos­si­ble, given the prac­ti­cal limi­ta­tions.

• You are rea­son­ing from the faulty as­sump­tion that “surely it’s pos­si­ble to for­mal­ize the prob­lem some­how and do some­thing”. The prob­lem state­ment is self-con­tra­dic­tory. We need to re­solve the con­tra­dic­tion. It’s only pos­si­ble by mak­ing some part of the prob­lem state­ment false. That’s what the prior over Omegas is for. We’ve been told some bul­lshit, and need to de­ter­mine which parts are true. Note how my Omegas of type 1 and 2 ban­ish the para­dox: in case 1 “the money is already there any­way” has be­come a plain sim­ple lie, and in case 2 “Omega has already pre­dicted your choice” be­comes a lie when you’re in­side Omega. I say the real world doesn’t have con­tra­dic­tions. Don’t ask me to rea­son ap­prox­i­mately from con­tra­dic­tory as­sump­tions.

• You gotta de­cide some­thing, faced with the situ­a­tion. It doesn’t look like you ar­gue that New­comb’s test it­self liter­ally can’t be set up. So what do you mean by con­tra­dic­tions? The phys­i­cal sys­tem it­self can’t be false, only its de­scrip­tion. What­ever con­tra­dic­tions you per­ceive in the test, they come from the prob­lems of in­ter­pre­ta­tion; the only rele­vant part of this whole en­deavor is com­put­ing the de­ci­sion.

• The phys­i­cal sys­tem can’t be false, but Omega seems to be ly­ing to us. How do you, as a ra­tio­nal­ist, deal when peo­ple con­tra­dict them­selves ver­bally? You build mod­els, like I did in the origi­nal post.

• Omega doesn’t lie by the state­ment of the prob­lem. It doesn’t even as­sert any­thing, it just places the money in the box or doesn’t.

• What’s wrong with you? If Omega tells us the con­di­tions of the ex­per­i­ment (about “foretel­ling” and stuff), then Omega is ly­ing. If some­one else, then some­one else. Let’s wrap this up, I’m sick.

• As was pointed out nu­mer­ous times, it well may be pos­si­ble to foretell your ac­tions, even by some vari­a­tion on just read­ing this fo­rum and look­ing what peo­ple claim to choose in the given situ­a­tion. That you came up with spe­cific ex­am­ples that ridicule the claim of be­ing able to pre­dict your de­ci­sion, doesn’t mean that there liter­ally is no way to do that. Another, more de­tailed ex­am­ple, is what you listed as (2) simu­la­tion ap­proach.

• some vari­a­tion on just read­ing this fo­rum and look­ing what peo­ple claim to choose in the given situation

Case 3, “ter­mi­nat­ing Omega”, demon­stra­ble con­tra­dic­tion.

Another, more de­tailed ex­am­ple, is what you listed as (2) simu­la­tion ap­proach.

I already ex­plained where a “simu­la­tor Omega” has to lie to you.

Sorry, I don’t want to spend any more time on this dis­cus­sion. Good­bye.

• FWIW, I un­der­stand your frus­tra­tion, but just as a data point I don’t think this re­ac­tion is war­ranted, and I say that as some­one who likes most of your com­ments. I know you made this post in or­der to es­cape the rab­bit hole, but you must have ex­pected to spend a lit­tle time there dig­ging when you made it!

• The prob­lem set­ting it­self shouldn’t raise many ques­tions. If you agree that the right an­swer in this set­ting is to one-box, you prob­a­bly un­der­stand the test. Next, look at the pop­u­lar de­ci­sion the­o­ries that calcu­late that the “cor­rect” an­swer is to two-box. Find what’s wrong with those the­o­ries, or with the ways of ap­ply­ing them, and find a way to gen­er­al­ize them to han­dle this case and other cases cor­rectly.

• There’s noth­ing wrong with those the­o­ries. They are wrongly ap­plied, se­lec­tively ig­nor­ing the part of the prob­lem state­ment that ex­plic­itly says you can’t two-box if Omega de­cided you would one-box. Any naive ap­pli­ca­tion will do that be­cause all stan­dard the­o­ries as­sume causal­ity, which is bro­ken in this prob­lem. Be­fore ap­ply­ing de­ci­sion the­o­ries we must work out what causes what. My origi­nal post was an at­tempt to do just that.

What other cases?

• There’s noth­ing wrong with those the­o­ries. They are wrongly ap­plied, se­lec­tively ig­nor­ing the part of the prob­lem state­ment that ex­plic­itly says you can’t two-box if Omega de­cided you would one-box.

The de­ci­sion is yours, Omega only fore­sees it. See also: Thou Art Physics.

Any naive ap­pli­ca­tion will do that be­cause the prob­lem state­ment is con­tra­dic­tory on the sur­face. Be­fore ap­ply­ing de­ci­sion the­o­ries, the con­tra­dic­tion has to be re­solved some­how as we work out what causes what. My origi­nal post was an at­tempt to do just that.

Do that for the stan­dard set­ting that I out­lined above, in­stead of con­struct­ing its bro­ken vari­a­tions. What it means for some­thing to cause some­thing else, and how one should go about de­scribing the situ­a­tions in that model should ar­guably be a part of any de­ci­sion the­ory.

• the prob­lem state­ment … ex­plic­itly says you can’t two-box if Omega de­cided you would one-box.

The de­ci­sion is yours, Omega only fore­sees it.

Th­ese stop con­tra­dict­ing each other if you rephrase a lit­tle more pre­cisely. It’s not that you can’t two-box if Omega de­cided you would one-box—you just don’t, be­cause in or­der for Omega to have de­cided that, you must have also de­cided that. Or rather, been go­ing to de­cide that—and if I un­der­stand the post you linked cor­rectly, its point is that the differ­ence be­tween “my de­ci­sion” and “the pre­de­ter­mi­na­tion of my de­ci­sion” is not mean­ingful.

As far as I can tell—and I’m new to this topic, so please for­give me if this is a ju­ve­nile ob­ser­va­tion—the flaw in the prob­lem is that it can­not be true both that the con­tents of the boxes are de­ter­mined by your choice (via Omega’s pre­dic­tion), and that the con­tents have already been de­ter­mined when you are mak­ing your choice. The ar­gu­ment for one-box­ing as­sumes that, of those con­tra­dic­tory premises, the first one is true. The ar­gu­ment for two-box­ing as­sumes that the sec­ond one is true.

The po­ten­tial flaw in my de­scrip­tion, in turn, is whether my sim­plifi­ca­tion just now (“de­ter­mined by your choice via Omega”) is ac­tu­ally equiv­a­lent to the way it’s put in the prob­lem (“de­ter­mined by Omega based on a pre­dic­tion of you”). I think it is, for the rea­sons given above, but what do I know?

(I feel com­fortable enough with this ex­pla­na­tion that I’m quite con­fi­dent I must be miss­ing some­thing.)

• An as­piring Bayesian ra­tio­nal­ist would be­have like me in the origi­nal post: as­sume some prior over the pos­si­ble im­ple­men­ta­tions of Omega and work out what to do. So taboo “fore­see” and pro­pose some mechanisms as I, ci­pher­goth and Toby Ord did.

• Why shouldn’t you ad­just your crite­ria for ap­proval un­til they fit the de­ci­sion the­ory?

• Why not ad­just both un­til you get a mil­lion dol­lars?

• I’m lik­ing this prefer­ence for (Zen|So­cratic) re­sponses.

• Thank you. Hope­fully this will be the last post about New­comb’s prob­lem for a long time.

Even dis­re­gard­ing un­cer­tainty whether you’re run­ning in­side Omega or in the real world, as­sum­ing Omega is perfect #2 effec­tively re­verses the or­der of de­ci­sions just like #1 - and you de­cide first (via simu­la­tion), omega de­cides sec­ond. So it col­lapses to a triv­ial one-box.

• taw, I was kinda hop­ing you’d have some al­ter­na­tive for­mu­la­tions, hav­ing thought of it longer than me. What do you think? Is it still pos­si­ble to res­cue the prob­lem?

• I was mostly try­ing to ap­proach it from clas­si­cal de­ci­sion the­ory side, but the re­sults are still the same. There are three lev­els in the de­ci­sion tree here:

• You pre­com­mit to one-box /​ two-box

• Omega de­cides 1000000 /​ 0. Omega is al­lowed to look at your precommitment

• You do one-box /​ two-box

If we con­sider pre­com­mit­ment to be bind­ing, we col­lapse it to “you de­cide first, omega sec­ond, so triv­ial one-box” . If we con­sider pre­com­mit­ment non-bind­ing, we col­lapse it to “you make throw­away de­ci­sion to one-box, omage does 1000000, you two-box and get 1001000″, and this “omega” has zero knowl­edge.

In clas­si­cal de­ci­sion the­ory you are not al­lowed to look at other peo­ple’s pre­com­mit­ments, so the game with de­ci­sions tak­ing place at any point (be­tween start and the ac­tion) and peo­ple chang­ing their minds on ev­ery step is math­e­mat­i­cally equiv­a­lent to one where pre­com­mit­ments are bind­ing and de­cided be­fore any­body acts.

This equiv­alency is bro­ken by New­comb’s prob­lem, so pre­com­mit­ments and be­ing able to break them now do mat­ter, and peo­ple who try to use clas­si­cal de­ci­sion the­ory ig­nor­ing this will fail. Ax­iom bro­ken, ev­ery­body dies.

• Omega simu­lates your de­ci­sion al­gorithm. In this case the de­ci­sion al­gorithm has in­dex­i­cal un­cer­tainty on whether it’s be­ing run in­side Omega or in the real world, and it’s log­i­cal to one-box thus mak­ing Omega give the “real you” the mil­lion.

I never thought of that!

Can you for­mal­ize “hilar­ity en­sues” a bit more pre­cisely?

• I’d love to claim credit, but the head-slap­ping idea was men­tioned on OB more than once, and also in the Wikipe­dia en­try on New­comb’s Para­dox.

Hilar­ity means we know what Omega pre­dicted but are free to do what we like. For ex­am­ple, you could learn that Omega con­sid­ers you a two-boxer and then one-box, earn­ing zero money—an im­pres­sive feat con­sid­er­ing the cir­cum­stances.

• It’s like a Master­card com­mer­cial. Los­ing the op­por­tu­nity to get a stack of money: costly. Blow­ing Omega’s mind: price­less.

• I love how the dis­cus­sion here is turn­ing out. The post had karma 1, then 0, then 1 again and there it stays; but the chat is quite lively. Maybe I shouldn’t ob­sess over karma.

• Sadly, it’s im­pos­si­ble to dis­t­in­guish a com­ment no one votes on from one that has equal pos­i­tive and nega­tive votes. The ‘most con­tro­ver­sial’ cat­e­gory op­tion helps a lit­tle bit, but not much.

My ad­vice: don’t sweat the small stuff, and re­mem­ber that votes are small stuff.

• Sadly, it’s im­pos­si­ble to dis­t­in­guish a com­ment no one votes on from one that has equal pos­i­tive and nega­tive votes.

This may get fixed later.

• Omega knows that I have no pa­tience for log­i­cal para­doxes, and will del­e­gate my de­ci­sion to a quan­tum coin-flip­per ex­ploit­ing the Con­way-Kochen the­o­rem. Hilar­ity en­sues.

• I would one-box in New­comb’s prob­lem, but I’m not sure why Omega is more plau­si­ble than a be­ing that re­wards peo­ple that it pre­dicts would be two-box­ers. And yet it is more plau­si­ble to me.

When I as­so­ci­ate one-box­ing with co­op­er­a­tion, that makes it more at­trac­tive. The anti-Omega would be some­one who was afraid co­op­er­a­tors would con­spire against it, and so it re­wards the op­po­site.

In the case of the pre-mi­graine state be­low, re­frain­ing from choco­late seems much less com­pel­ling.

• 4) Same as 3, but the uni­verse only has room for one Omega, e.g. the God Almighty. Then ipso facto it can­not ever be mod­el­led math­e­mat­i­cally, and let’s talk no more.

Why can’t God Almighty be mod­el­led math­e­mat­i­cally?

Omega/​God is run­ning the uni­verse on his com­puter. He can pause any time he wants (for ex­am­ple to run some calcu­la­tions), and mod­ify the “uni­verse state” to com­mu­ni­cate (or just put his boxes in).

That seems to be close enough to 4). Un­like with 3), you can’t use the same pro­cess as Omega (pause the uni­verse and run ar­bi­trary calcu­la­tions that could con­sider the state of ev­ery quark).

• No God Almighty needed for your ex­am­ple, just an in­tel­li­gence that’s defined to be more pow­er­ful than you. If your com­pu­ta­tional ca­pac­ity is bounded and the other player has much more, you cer­tainly can’t ap­ply any perfectly ra­tio­nal de­ci­sion con­cept. The prob­lem is now about ap­prox­i­ma­tion. One ap­prox­i­ma­tion I’ve men­tioned sev­eral times already is be­liev­ing pow­er­ful agents with a 100% track record of truth. Sound rea­son­able? That’s the level of dis­cus­sion you get when you in­tro­duce bounds.

• Your Omega isn’t a type 3 or 4 at all, it’s a type 2 with re­ally big com­pu­ta­tional ca­pac­ity.

• What does New­comb’s Prob­lem has to do with re­al­ity as we know it any­way? I mean, imag­ine that I’ve solved it (what­ever that means). Where in my ev­ery­day life can I ap­ply it?

• Parfit’s Hitch­hiker, col­lid­ing fu­tur­is­tic civ­i­liza­tions, AIs with knowl­edge of each other’s source code, whether ra­tio­nal­ists can in prin­ci­ple co­op­er­ate on the true Pri­soner’s Dilemma.

• Oh, hello.

Parfit’s Hitchhiker

Purely about pre­com­mit­ment, not pre­dic­tion. Precom­mit­ment has been an­a­lyzed to death by Schel­ling, no para­doxes there.

col­lid­ing fu­tur­is­tic civilizations

Pass.

AIs with knowl­edge of each other’s source code

Rice’s the­o­rem.

whether ra­tio­nal­ists can in prin­ci­ple co­op­er­ate on the true Pri­soner’s Dilemma

PD doesn’t have mys­ti­cal om­ni­scient en­tities. If we try to elimi­nate them from New­comb’s as well, the prob­lem evap­o­rates. So no re­la­tion.

• Rice’s the­o­rem.

You keep us­ing that word. I do not think it means what you think it does.

• Rice’s the­o­rem is ev­i­dence that Omega is likely to be type 1 or 2 rather than 3, and thus in fa­vor of one-box­ing.

• This was kinda the point of the post: demon­strate the craz­i­ness and ir­rele­vance of the prob­lem. I just got sick of peo­ple here cit­ing it as an im­por­tant ex­am­ple. The eas­iest way to dis­miss a prob­lem like that from our col­lec­tive mind is to “solve” it.

• Parfit’s Hitch­hiker, col­lid­ing fu­tur­is­tic civ­i­liza­tions, AIs with knowl­edge of each other’s source code, whether ra­tio­nal­ists can in prin­ci­ple co­op­er­ate on the true Pri­soner’s Dilemma.

• I have a very strong feel­ing that way 3 is not pos­si­ble. It seems that any scan­ning/​anal­y­sis pro­ce­dure de­tailed enough to pre­dict your ac­tions con­sti­tutes simu­lat­ing you.

• I have a very strong feel­ing that way 3 is not pos­si­ble. It seems that any scan­ning/​anal­y­sis pro­ce­dure de­tailed enough to pre­dict your ac­tions con­sti­tutes simu­lat­ing you.

I pre­dict that you will not, in the next 24 hours, choose to com­mit suicide.

Am I simu­lat­ing you?

• To com­plete the pic­ture you should give smoofra ad­e­quate in­cen­tive to falsify your pre­dic­tion, and then see how it goes.

• You can always change the prob­lem so that it stops mak­ing sense, or that the an­swer gets re­versed. But this is not the point, you should seek to un­der­stand what the in­tent was as clearly as pos­si­ble.

If an ar­gu­ment at­tacks your long-held be­lief, make the ar­gu­ment stronger, help it to get through. If you were right, the ar­gu­ment will fail, but you ought to give it the best chance you can.

• Not nec­es­sar­ily. It could be purely em­piri­cal in na­ture. No in­sight into how the de­tected sig­nals causally re­late to the out­put is re­quired.

• I feel the same, but would have been dishon­est to omit it. Even 4 sounds more likely to me than 3.