# The Doubling Box

Let’s say you have a box that has a to­ken in it that can be re­deemed for 1 utilon. Every day, its con­tents dou­ble. There is no limit on how many utilons you can buy with these to­kens. You are im­mor­tal. It is sealed, and if you open it, it be­comes an or­di­nary box. You get the to­kens it has cre­ated, but the box does not dou­ble its con­tents any­more. There are no other ways to get utilons.

How long do you wait be­fore open­ing it? If you never open it, you get noth­ing (you lose! Good day, sir or madam!) and when­ever you take it, tak­ing it one day later would have been twice as good.

I hope this doesn’t sound like a re­duc­tio ad ab­sur­dum against un­bounded util­ity func­tions or not dis­count­ing the fu­ture, be­cause if it does you are in dan­ger of am­pu­tat­ing the wrong limb to save your­self from para­dox-gan­grene.

What if in­stead of grow­ing ex­po­nen­tially with­out bound, it de­cays ex­po­nen­tially to the bound of your util­ity func­tion? If your util­ity func­tion is bounded at 10, what if the first day it is 5, the sec­ond 7.5, the third 8.75, etc. As­sume all the lit­tle de­tails, like re­mem­ber­ing about the box, trad­ing in the to­kens, etc, are free.

If you dis­count the fu­ture us­ing any func­tion that doesn’t ever hit 0, then the growth rate of the to­kens can be cho­sen to more than make up for your dis­count­ing.

If it does hit 0 at time T, what if in­stead of dou­bling, it just in­creases by how­ever many utilons will be ad­justed to 1 by your dis­count­ing at that point ev­ery time of growth, but the in­ter­vals of growth shrink to noth­ing? You get an ad­justed 1 utilon at time T − 1s, and an­other ad­justed 1 utilon at T − 0.5s, and an­other at T − 0.25s, etc? Sup­pose you can think as fast as you want, and open the box at ar­bi­trary speed. Also, that what­ever solu­tion your pre­sent self pre­com­mits to will be fol­lowed by the fu­ture self. (Their de­ci­sion won’t be changed by any change in what times they care about)

EDIT: Peo­ple in the com­ments have sug­gested us­ing a util­ity func­tion that is both bounded and dis­count­ing. If your util­ity func­tion isn’t so strongly dis­count­ing that it drops to 0 right af­ter the pre­sent, then you can find some time in­ter­val very close to the pre­sent where the dis­count­ing is all nonzero. And if it’s nonzero, you can have a box that dis­ap­pears, tak­ing all pos­si­ble util­ity with it at the end of that in­ter­val, and that, lead­ing up to that in­ter­val, grows the util­ity in in­ter­vals that shrink to noth­ing as you ap­proach the end of the in­ter­val, and in­creas­ing the util­ity-worth of to­kens in the box such that it com­pen­sates for what­ever your dis­count­ing func­tion is ex­actly enough to asymp­tot­i­cally ap­proach your bound.

Here is my solu­tion. You can’t as­sume that your fu­ture self will make the op­ti­mal de­ci­sion, or even a good de­ci­sion. You have to treat your fu­ture self as a phys­i­cal ob­ject that your choices af­fect, and take the prob­a­bil­ity dis­tri­bu­tion of what de­ci­sions your fu­ture self will make, and how much util­ity they will net you into ac­count.

Think if your­self as a Tur­ing ma­chine. If you do not halt and open the box, you lose and get noth­ing. No mat­ter how com­pli­cated your brain, you have a finite num­ber of states. You want to be a busy beaver and take the most pos­si­ble time to halt, but still halt.

If, at the end, you say to your­self “I just counted to the high­est num­ber I could, count­ing once per day, and then made a small mark on my skin, and re­peated, and when my skin was full of marks, that I was con­stantly re­fresh­ing to make sure they didn’t go away...

...but I could let it dou­ble one more time, for more util­ity!”

If you re­turn to a state you have already been at, you know you are go­ing to be wait­ing for­ever and lose and get noth­ing. So it is in your best in­ter­est to open the box.

So there is not a uni­ver­sal op­ti­mal solu­tion to this prob­lem, but there is an op­ti­mal solu­tion for a finite mind.

I re­mem­ber read­ing a while ago about a para­dox where you start with \$1, and can trade that for a 50% chance of \$2.01, which you can trade for a 25% chance of \$4.03, which you can trade for a 12.5% chance of \$8.07, etc (can’t re­mem­ber where I read it).

This is the same para­dox with one of the traps for wannabe Cap­tain Kirks (us­ing dol­lars in­stead of utilons) re­moved and one of the un­nec­es­sary vari­ables (un­cer­tainty) cut out.

My solu­tion also works on that. Every trade is analo­gous to a day waited to open the box.

• This prob­lem makes more sense if you strip out time and the dou­bling, and look at this one:

Choose an in­te­ger N. Re­ceive N utilons.

This prob­lem has no op­ti­mal solu­tion (be­cause there is no largest in­te­ger). You can com­pare any two strate­gies to each other, but you can­not find a supre­mum; the clos­est thing available is an in­finite se­ries of suc­ces­sively bet­ter strate­gies, which even­tu­ally passes any sin­gle strat­egy.

In the origi­nal prob­lem, the op­tions are “don’t open the box” or “wait N days, then open the box”. The former can be crossed off; the lat­ter has the same in­finite se­ries of suc­ces­sively bet­ter strate­gies. (The ap­par­ent time-sym­me­try is a false one, be­cause there are only two time-in­var­i­ant strate­gies, and they both lose.)

The way to solve this in de­ci­sion the­ory is to ei­ther in­tro­duce finite­ness some­where that caps the num­ber of pos­si­ble strate­gies, or to out­put an or­der­ing over choices in­stead of a sin­gle choice. The lat­ter seems right; if you define and prove an in­finite se­quence of suc­ces­sively bet­ter op­tions, you still have to pick one; and lat­tices seem like a good way to rep­re­sent the re­sults of par­tial rea­son­ing.

• This is pretty much the only com­ment in the en­tire thread that doesn’t fight the hy­po­thet­i­cal. Well done, I guess?

• This seems like a helpful sim­plifi­ca­tion of the prob­lem. Note that it also works if you re­ceive 1-1/​N utilons, so as with the origi­nal post this isn’t an un­bounded util­ity is­sue as such.

Just one point though—in the origi­nal prob­lem speci­fi­ca­tion it’s ob­vi­ous what “choose an in­te­ger N” means: open­ing a phys­i­cal box on day n cor­re­sponds to choos­ing 2^n. But how does your prob­lem get em­bed­ded in re­al­ity? Do you need to write your cho­sen num­ber down? As­sum­ing there’s no time limit to writ­ing it down then this be­comes very similar to the origi­nal prob­lem ex­cept you’re mul­ti­ply­ing by 10 in­stead of 2 and the time in­ter­val is the time taken to write an ex­tra digit in­stead of a day.

• But how does your prob­lem get em­bed­ded in re­al­ity?

It doesn’t. It gets em­bed­ded in some­thing with in­finite time and in which in­finite util­ity can be given out (in in­finite differ­ent de­nom­i­na­tions).

• … to write an ex­tra digit …

Writ­ing dec­i­mal digits isn’t the op­ti­mal way to write big num­bers. (Of course this doesn’t in­val­i­date your point.)

• It kind of is if you have to be able to write down any num­ber.

• I’ve never met an in­finite de­ci­sion tree in my life so far, and I doubt I ever will. It is a prop­erty of prob­lems with an in­finite solu­tion space that they can’t be solved op­ti­mally, and it doesn’t re­veal any de­ci­sion the­o­retic in­con­sis­ten­cies that could come up in real life.

Con­sider this game with a tree struc­ture: You pick an ar­bi­trary nat­u­ral num­ber, and then, your op­po­nent does as well. The player who chose the high­est num­ber wins. Clearly, you can­not win this game, as no mat­ter which num­ber you pick, the op­po­nent can sim­ply add one to that num­ber. This also works with pick­ing a pos­i­tive ra­tio­nal num­ber that’s clos­est to 1 - your op­po­nent here adds one to the de­nom­i­na­tor and the nu­mer­a­tor, and wins.

The idea to use a busy beaver func­tion is good, and if you can uti­lize the en­tire uni­verse to en­code the states of the busy beaver with the largest num­ber of states pos­si­ble (and a long enough tape), then that con­sti­tutes the op­ti­mal solu­tion, but that only takes us fur­ther out into the realm of fic­tion.

• I’ve never met an in­finite de­ci­sion tree in my life so far, and I doubt I ever will. It is a prop­erty of prob­lems with an in­finite solu­tion space that they can’t be solved op­ti­mally, and it doesn’t re­veal any de­ci­sion the­o­retic in­con­sis­ten­cies that could come up in real life.

“You are finite. Zathras is finite. This util­ity func­tion has in­fini­ties in it. No, not good. Never use that.”
— Not Baby­lon 5

• But I do not choose my util­ity func­tion as an means to get some­thing. My util­ity func­tion de­scribes is what I want to choose means to get. And I’m pretty sure it’s un­bounded.

• You’ve only ex­pended a finite amount of com­pu­ta­tion on the ques­tion, though; and you’re run­ning on cor­rupted hard­ware. How con­fi­dent can you be that you have already cor­rectly dis­t­in­guished an un­bounded util­ity func­tion from one with a very large finite bound?

(A geno­ci­dal, fa­nat­i­cal ass­hole once said: “I be­seech you, in the bow­els of Christ, think it pos­si­ble that you may be mis­taken.”)

• I do think it pos­si­ble I may be mis­taken.

• I like this. I was go­ing to say some­thing like,

“Sup­pose , what does that say about your solu­tions de­signed for real life?” and screw you I hate when peo­ple do this and think it is clever. Utility mon­ster is an­other ex­am­ple of this sort of non­sense.

but you said the same thing, and less rudely, so up­voted.

• The tax man’s dilemma, an in­finite de­ci­sion tree grounded in re­al­ity:

As­sume you’re the an­thro­po­mor­phiza­tion of gov­ern­ment. And you have a de­ci­sion to make: You need to de­cide the ideal tax rate for busi­nesses.

In your so­ciety, cor­po­ra­tions re­li­ably make 5% re­turns on in­vest­ments, ac­count­ing for in­fla­tion. That money is re­li­ably rein­vested, al­though not nec­es­sar­ily in the same cor­po­ra­tion.

How should you tax those re­turns in or­der to max­i­mize to­tal util­ity? You may change taxes at any point. Also, you’re the an­thro­po­mor­phic rep­re­sen­ta­tion of gov­ern­ment—you are, for all in­tents and pur­poses, im­mor­tal.

As­sume a fu­ture util­ity dis­count rate of less than the in­vest­ment re­turn rate, and as­sume you don’t know the money-util­ity re­la­tion­ship—you can say you weigh the pos­si­bil­ity of fu­ture dis­asters which re­quire im­mense funds against the pos­si­bil­ity that money has de­clin­ing util­ity over time to pro­duce a con­stant re­la­tion­ship for sim­plic­ity, if you wish. As­sume that your re­turns will be less than cor­po­rate re­turns, and cor­po­rate util­ity will be less than your util­ity. (Sim­plified, you pro­duce no in­vest­ment re­turns, cor­po­ra­tions pro­duce no util­ity.)

• I never saw an in­finite de­ci­sion tree
I doubt I’ll ever see one
But if it weren’t imag­i­nary
I’d rather like to play one

Hmm I guess I see now why so few clas­sic po­ets used de­ci­sion the­ory as their in­spira­tion.

• If it’s im­pos­si­ble to win, be­cause your op­po­nent always picks sec­ond, then ev­ery choice is op­ti­mal.

If you pick si­mul­ta­neously, pick­ing the high­est num­ber you can de­scribe is op­ti­mal, so that’s an­other situ­a­tion where there is no op­ti­mal solu­tion for an in­finite mind, but for a finite mind, there is an op­ti­mal solu­tion.

• 6 Aug 2012 14:05 UTC
14 points

After con­sid­er­ing this prob­lem, what I found was sur­pris­ingly fast, the speci­fics of the boxes phys­i­cal abil­ities and im­ple­men­ta­tion be­comes rele­vant. I mean, let’s say Clippy is given this box, and has already de­cided to wait a mere 1 year from day 1, which is 365.25 days of dou­bling, and 1 pa­per­clip is 1 utilon. At some point, dur­ing this time, be­fore the end of it, There are more pa­per­clips then there used to be ev­ery atom in the visi­ble uni­verse. Since he’s pre­dicted to gain 2^365.25 pa­per­clips, (which is ap­par­ently close to 8.9*10^109) and the ob­serv­able uni­verse is only es­ti­mated to con­tain 10^80 atoms. So to make up for that, let’s say the box con­verts ev­ery visi­ble sub­atomic par­ti­cle into pa­per­clips in­stead.

That’s just 1 year, and the box has already an­nounced it will con­vert ap­prox­i­mately ev­ery visi­ble sub­atomic par­ti­cle into pure pa­per­clip bliss!

And then an­other sin­gle dou­bling… (1 year and 1 day) Does what? Even if Clippy has his util­ity func­tion un­bounded, it should pre­sum­ably still link back to some kind of phys­i­cal state, and at this point the box starts hav­ing to im­ple­ment in­creas­ingly phys­i­cally im­pos­si­ble ideas to have to dou­ble pa­per­clip util­ity, like:

Break­ing the speed of light.

Ex­pand­ing the pa­per­clip con­ver­sion into the past.

Ex­pand­ing the pa­per­clip con­ver­sion into ad­di­tional branches of many wor­lds.

Mag­i­cally pro­tect­ing the pa­per­clips from the rav­ages of time, physics, or con­dens­ing into black­holes, de­spite the fact it is sup­posed to lose all power af­ter be­ing opened.

And that’s just 1 year! We aren’t even close to a time­less eter­nity of wait­ing yet, and the box already has to smash the cur­rently known laws of physics (more so than it did by con­vert­ing ev­ery visi­ble sub­atomic par­ti­cle into pa­per­clips) to do more dou­blings, and will then lose power af­ter­wards.

Do the laws of physics re­sume be­ing nor­mal af­ter the box loses power? If so, mas­sive chunks of util­ity will fade away al­most in­stantly (which would seem to in­di­cate the Box was not very effec­tive), but if not I’m not sure how the loop be­low would get re­solved:

The Box is es­sen­tially go­ing to rewrite the rules of the uni­verse per­ma­nently,

Which would af­fect your util­ity calcu­la­tions, which are based on physics,

Which would af­fect how the Box rewrote the rules of the uni­verse,

Which would af­fect your util­ity calcu­la­tions, which are based on physics,

Ex­cept in­stead of stop­ping and let­ting you and the box re­solve this loop, it must keeps dou­bling, so it keeps chang­ing physics more.

By year 2, it seems like you might be left with ei­ther:

A solu­tion, in which case what­ever the box will rewrite the laws of physics to, you un­der­stand and agree with it and can work on the prob­lem based on what­ever that solu­tion is. (But I have no idea how you could figure out what this solu­tion would be in ad­vance, since it de­pends on the spe­cific box?)

Or, an in­cred­ibly in­tractable meta­physics prob­lem which is grow­ing more com­pli­cated faster than you can ever calcu­late, in which case you don’t even un­der­stand what the box is do­ing any­more.

The rea­son I said that this was in­cred­ibly fast is that my origi­nal guess was that it would take at least 100 years of daily dou­bling for the pro­posed world to be­come that com­pli­cated, but when I tried do­ing a bit of math it didn’t take any­where near that long.

Edit: Fixed a few ty­pos and cleared up gram­mar.

• This is a thought ex­per­i­ment which is not meant to be pos­si­ble in our world. But such thought ex­per­i­ments are a way of test­ing the gen­er­al­ity of your de­ci­sion pro­ce­dures—do they work in all pos­si­ble wor­lds? If you must imag­ine a physics that makes the eter­nal dou­bling pos­si­ble, try pic­tur­ing a net­work of repli­cat­ing baby uni­verses linked by worm­holes.

• But such thought ex­per­i­ments are a way of test­ing the gen­er­al­ity of your de­ci­sion pro­ce­dures—do they work in all pos­si­ble wor­lds?

As in the old saw, part of your strength as a real de­ci­sion-maker is that your de­ci­sion pro­ce­dures choose less well in im­pos­si­ble wor­lds than in pos­si­ble wor­lds.

• A world that can sup­port pa­per­clip pro­duc­tion of ar­bi­trary mag­ni­tude is not an im­pos­si­ble world. The speed of light is a con­tin­gent fact.

• Why does that have to be true?

• It doesn’t have to be true. It’s de­sir­able be­cause de­ci­sion pro­ce­dures that rely on other knowl­edge about re­al­ity are faster/​bet­ter/​cheaper than ones that don’t im­port knowl­edge about re­al­ity. Spe­cial­iza­tion for the situ­a­tion you find your­self in is of­ten use­ful, though it does limit flex­i­bil­ity.

• Utility doesn’t have to be pro­por­tional to the amount of some par­tic­u­lar kind of phys­i­cal stuff in the uni­verse. If the uni­verse con­tained 1 pa­per­clip, that could be worth 2 utilons, if it con­tained 2 pa­per­clips then it could be worth 4 utilons, if it con­tained 20 pa­per­clips then it could be worth 2^20 utilons. The box would then dou­ble your util­ity each day just by adding one phys­i­cal pa­per­clip.

I still think these kinds of con­sid­er­a­tions are worth think­ing about though. Your util­ity func­tion might grow faster than a busy beaver func­tion, but then the dou­bling box is go­ing to have trou­ble wait­ing the right length of time to de­liver the

• Your other op­tion is to sell the box to the high­est bid­der. That will prob­a­bly be some­one who’s pre­pared to wait longer than you, and will there­fore be able to give you a higher price than the utilons you’d have got out of the box your­self. You get the utilons to­day.

• I don’t think you are a wannabe Cap­tain Kirk. I think you are Cap­tain Kirk.

• Why does my fight-the-hy­po­thet­i­cal mod­ule never think about that? (It does of­ten think about op­tions which wouldn’t be available in the Least Con­ve­nient Pos­si­ble World—but not this one, un­til some­one else points it out.)

• If you can use mixed strate­gies (i.e. are not re­quired to be de­ter­minis­ti­cally pre­dictable), you can use the fol­low­ing strat­egy for the dou­bling-util­ity case: ev­ery day, toss a coin; if it comes up heads, open the box, oth­er­wise wait an­other day. Ex­pected util­ity of each day is con­stant 12, since the prob­a­bil­ity of get­ting heads on a par­tic­u­lar day halves with each sub­se­quent day, and util­ity dou­bles, so the se­ries di­verges and you get in­finite to­tal ex­pected util­ity.

• Even bet­ter, how­ever, would be to toss two coins ev­ery day, and only open the box if both come up heads :)

• This sug­gests a strat­egy; tile the uni­verse with coins and flip each of them ev­ery day. If they all come up heads, open the box (pre­sum­ably it’s full of even more coins).

• bet­ter yet, ev­ery day count one more in­te­ger to­ward the high­est num­ber you can think of, when you reach it, flip the coins. If they don’t all come up heads, start over again.

• That way your ex­pected util­ity be­comes INFINITY TIMES TWO! :)

• There are mean­ingful ways to com­pare two out­comes which both have in­finite ex­pected util­ity. For ex­am­ple, sup­pose X is your fa­vorite in­finite-ex­pected-util­ity out­come. Then a 20% chance of X (and 80% chance of noth­ing) is bet­ter than a 10% chance of X. Some­thing similar hap­pens with toss­ing two coins in­stead of one, al­though it’s more sub­tle.

• Ac­tu­ally what you get is an­other di­ver­gent in­finite se­ries that grows faster. They both grow ar­bi­trar­ily large, but the one with p=0.25 grows ar­bi­trar­ily larger than the se­ries with p=0.5, as you com­pute more terms. So there is no sense in which the sec­ond se­ries is twice as big, al­though there is a sense in which it is in­finitely larger. (I know your point is that they’re both tech­ni­cally the same size, but I think this is worth not­ing.)

• This is what I was go­ing to say; it’s con­sis­tent with the ap­par­ent time sym­me­try, and is the only solu­tion that makes sense if we ac­cept the prob­lem as stated. But it seems like the wrong an­swer in­tu­itively, be­cause it means that ev­ery strat­egy is equal, as long as the prob­a­bil­ity of open­ing the box on a given day is in the half-open in­ter­val (0,0.5]. I’d cer­tainly be hap­pier with, say, p=0.01 than p=0.5, (and so would ev­ery­one else, ap­par­ently) which sug­gests that I don’t ac­tu­ally have a real-val­ued util­ity func­tion. This might be a good ar­gu­ment against real-val­ued util­ity func­tions in gen­eral (bounded or not). Espe­cially since a lot of the pro­posed solu­tions here “fight the hy­po­thet­i­cal” by point­ing out that real agents can only choose from a finite set of strate­gies.

• So you can have in­finite ex­pected util­ity, but be guaran­teed to have finite util­ity? That is weird.

• How ex­actly do the con­stant utilons in the box com­pen­sate me for how I feel the day af­ter I open the box (I could have dou­bled my cur­rent util­ity!)? The sec­ond day af­ter (I could have quadru­pled my cur­rent util­ity!!)? The Nth day af­ter (FFFFFFFFFFFFUUUUU!!!)? I’m afraid the box will rewrite me with a sim­ple rou­tine that says “I have 2^(day-I-opened-the-box − 1) util­ity! Yay!”

• What if in­stead of grow­ing ex­po­nen­tially with­out bound, it de­cays ex­po­nen­tially to the bound of your util­ity func­tion?

I think you mean ‘asymp­tot­i­cally’.

• Ex­po­nen­tially de­cay­ing func­tions are part of the gen­eral class of asymp­totic func­tions; both ver­sions are cor­rect.

• So I don’t re­ally know how utilons work, but here is an ex­am­ple of a util­ity func­tion which is dou­bling box-proof. It is bounded; fur­ther­more, it dis­counts the fu­ture by chang­ing the bound for things that only af­fect the fu­ture. So you can get up to 1000 utilons from some­thing that hap­pens to­day, up to 500 utilons from some­thing that hap­pens to­mor­row, up to 250 utilons from some­thing that hap­pens two days from now, and so on.

Then the solu­tion is ob­vi­ous: if you open the box in 4 days, you get 16 utilons; if you open the box in 5 days, you’d get 32 but your util­ity func­tion is bounded so you only get 31.25; if you open the box in 6 days or more, the re­ward just keeps shrink­ing. So 5 days is the best choice, as­sum­ing you can pre­com­mit to it.

A suit­able trans­for­ma­tion of this func­tion prob­a­bly does cap­ture the way we think about money: the util­ity of it is both bounded and dis­counted.

• So I don’t re­ally know how utilons work, but here is an ex­am­ple of a util­ity func­tion which is dou­bling box-proof. It is bounded; fur­ther­more, it dis­counts the fu­ture by chang­ing the bound for things that only af­fect the fu­ture. So you can get up to 1000 utilons from some­thing that hap­pens to­day, up to 500 utilons from some­thing that hap­pens to­mor­row, up to 250 utilons from some­thing that hap­pens two days from now, and so on.

You are right that such a util­ity func­tion can­not be sup­plied with a (util­ity) dou­bling box prob­lem—and for much the same rea­son that most util­ity func­tions that ap­prox­i­mate hu­man prefer­ences could not be ex­posed to a dou­bling box. Nev­er­the­less this amounts to re­fus­ing to en­gage with the game the­ory ex­am­ple rather than re­spond­ing to it.

• This thwarts the origi­nal box, but I just ed­ited the OP to de­scribe an­other box that would get this util­ity func­tion in trou­ble.

• So you’re sug­gest­ing, in my ex­am­ple, a box that ap­proaches 500 utilons over the course of a day, then dis­ap­pears?

This isn’t even a prob­lem. I just need to have a re­ally good re­ac­tion time to open it as close to 24 hours as pos­si­ble. Although at some point I may de­cide that the risk of miss­ing the time out­weighs the in­crease in utilons. Any­way this isn’t even a con­tro­ver­sial thought ex­per­i­ment in that case.

• I thought you would re­al­ize I was as­sum­ing what I did for the case with the util­ity func­tion that dis­counts com­pletely af­ter a cer­tain time: “Sup­pose you can think as fast as you want, and open the box at ar­bi­trary speed.”

But if your util­ity func­tion dis­counts based on the amount of think­ing you’ve done, not on time, I can’t think of an analo­gous trap for that.

• So, ideally, these util­ity func­tions wouldn’t be ar­bi­trary, but would some­how re­flect things peo­ple might ac­tu­ally think. So, for ex­am­ple, if the box is only al­lowed to con­tain vary­ing amounts of money, I would want to dis­count based on time (for rea­sons of in­vest­ment if noth­ing else) and also put an up­per bound on the util­ity I get (be­cause at some point you just have so much money you can af­ford pretty much any­thing).

When ar­bi­trary utilons get mixed in, it be­comes com­pli­cated, be­cause I dis­count differ­ent ways to get util­ity at a differ­ent rate. For in­stance, a cure for can­cer would be worth­less 50 years from now if peo­ple figured out how to cure can­cer in the mean­time already, at which point you’d to­tal up all the ca­su­alties from now un­til then and dis­count based on those. This is differ­ent from money be­cause even get­ting a dol­lar 100 years from now is not en­tirely pointless.

On the other hand, I don’t think my util­ity func­tion dis­counts based on the amount of think­ing I’ve done, at least not for money. I want to figure out what my true re­sponse to the prob­lem is, in that case (which is ba­si­cally equiv­a­lent to the “You get \$X. What do you want X to be?” prob­lem). I think it’s that af­ter I’ve spent a lot of time think­ing about it and de­cided X should be, say, 100 quadrillion, which gets me 499 utilons out of a max­i­mum of 500, then mak­ing the de­ci­sion and not think­ing about it more might be worth more than 1 utilon to me.

• Now you’re just dodg­ing the thought ex­per­i­ment by chang­ing it.

• In a way, yes. I’m try­ing to cleanly sep­a­rate the bits of the thought ex­per­i­ment I have an an­swer to, from the bits of the thought ex­per­i­ment I don’t have an an­swer to.

• I way to think about this prob­lem to put you in near mode is to imag­ine what the util­ity might look like. Ex:

Day 1: Find­ing a quar­ter on the ground

Day 2: A child in Africa get­ting \$5

.....

Day X: Cur­ing cancer

Dax X+1: Cur­ing can­cer, Alzheimers, and AIDS.

On one hand, by wait­ing a day, more peo­ple would die of can­cer. On the other, by not wait­ing, you’d doom all those fu­ture peo­ple to die of AIDS and Alzheimers.

• You are adding a con­di­tion that was not pre­sent in the origi­nal prob­lem. Namely, that ev­ery day you do not open the box, you lose some num­ber of util­ions.

• Whoops, you’re right.

• Sup­pose in­stead of mul­ti­ply­ing the util­ity by 2 each day, the box mul­ti­plied the util­ity by 1. Would it look like this?

Day 1: Cur­ing cancer

Day 2: Cur­ing cancer

Day 3: Cur­ing can­cer …

Prob­a­bly not—each of those “cur­ing can­cer” out­comes is not iden­ti­cal (can­cer gets cured on a differ­ent day) so you’d as­sign them differ­ent util­ities. In or­der to con­form to the speci­fi­ca­tion, the box would have to add an ex­tra sweet­ener each day in or­der to make up for a day’s worth of can­cer deaths.

• Am I cor­rect in as­sess­ing that your solu­tion is to stop when you can no longer com­pre­hend the value in the box? That is, when an ad­di­tional dou­bling has no sub­jec­tive mean­ing to you? (Un­til that point, you’re not in a state loop, as the value with each dou­bling pro­vides an in­put you haven’t en­coun­tered be­fore.)

I was about to sug­gest stop­ping when you have more utilons than your brain has states (pro­vided you could mea­sure such), but then it oc­curred to me the solu­tions might be analo­gous, even if they ar­rive at differ­ent num­bers.

• I wouldn’t want to stop when I couldn’t com­pre­hend what was in the box. I always want more util­ity, whether I can un­der­stand that I have it or not. My solu­tion is to wait as long as you can be­fore wait­ing any longer puts you in an in­finite loop and guaran­tees you will never get any.

• As long as you com­pre­hend the num­ber in the box, you’re not in an in­finite loop. The in­put is differ­ent. Once the num­ber is no longer mean­ingful, you’re po­ten­tially in an in­finite loop; the in­put is the same.

• I’m pretty sure I could stay out of an in­finite loop much longer than I could com­pre­hend what was in the box. The con­tents of the box are grow­ing ex­po­nen­tially with the num­ber of days. If I just count the num­ber of days, I can stay in the realm of small num­bers much longer.

• I wait un­til there are so many util­tons in the box that I can use them to get two iden­ti­cal boxes and have some util­tions left over. Every time a box has more than enough util­i­tons to make two iden­ti­cal boxes, I re­peat that step. Any util­i­tons not used to make new boxes are the div­i­dend of the in­vest­ment.

• Now that you men­tion that, that’s true, and it gives me sev­eral other weird ideas. The box gives you to­kens that you ex­change for utilons, which seem like they are sup­posed to be defined as “What­ever you want/​define them to be, based on your val­ues.”

Ergo, imag­ine a Happy Michaelos that gets about twice as much pos­i­tive utilons from ev­ery­thing com­pared to Sad Michaelos. Sad Michaelos gets twice as much NEGATIVE utilons from ev­ery­thing com­pared to Happy Michaelos.

Let’s say a cookie grants Happy Michaelos 1 utilons. It would take two cook­ies to grant Sad Michaelos 1 utilons. Let’s say a stubbed toe grants Sad Michaelos −1 utilons. It would take two stubbed toes to grant Happy Michaelos −1 utilons.

So if Happy Michaelos or Sad Michaelos gets to open the box and they are friends who sub­stan­tially share util­ity and cook­ies… It should be Sad Michaelos who does so (both will get more cook­ies that way.)

As far as I can tell, this is a rea­son­able in­ter­pre­ta­tion of the box.

So, I should prob­a­bly figure out how the peo­ple be­low would work, since they are in­creas­ingly un­rea­son­able in­ter­pre­ta­tions of the box:

Is es­sen­tially 1 mil­lion times worse off than Sad Michaelos. Ergo, it the logic above holds, Ex­tremely Sad Michaelos gets 2 mil­lion cook­ies from turn­ing in a sin­gle to­ken.

Hyper Pout Michaelos:

Is es­sen­tially 1 billion times worse off than Sad Michaelos. He also has a note in his util­ity func­tion that he will re­ceive -in­finity(aleph 0) utilons if he does not change his util­ity func­tion back to Sad Michaelos’s util­ity func­tion within 1 sec­ond af­ter the box is pow­er­less and he has con­verted all of his to­kens. If the logic above holds, Hyper Pout Michaelos gets 1 billion times more cook­ies than Sad Michaelos, and then gets to en­joy sub­stan­tially more utilons from them!

Om­nide­spairing Michaelos:

Is al­most im­pos­si­ble to grant utilons to. The cer­tainty of om­nipo­tence grants him 1 utilon. Every­thing else that might be pos­i­tive (say, a 99% chance of om­nipo­tence) grants him 0 utilons.

This is a co­her­ent util­ity func­tion. You can even live and have a nor­mal life with it if you also want to avoid nega­tive utilons (eat­ing might only grant -in­finite (aleph 0) utilons and not eat­ing might grant -in­finite (aleph 1) utilons.

Box Cyn­i­cal De­s­pair­max Michaelos:

Gets some aleph of nega­tive in­finite utilons from ev­ery de­ci­sion what­so­ever. Again, he can make de­ci­sions and go through­out the day, but any num­ber of the to­kens that the box grants don’t seem to map to any­thing rele­vant on his util­ity func­tion. For in­stance, wait­ing a day might cost him—in­finite (aleph 2) utilons. Ad­ding a finite num­ber of utilons is ir­rele­vant. He im­me­di­ately opens the box so he can dis­card the use­less to­kens and get back to avoid­ing the in­com­pre­hen­si­ble hor­rors of life, and this is (as far as I can tell) a cor­rect an­swer for him.

It seems like at least some of the util­ity func­tions above cheat the box, but I’m not sure which ones go to far, if the sam­ple is rea­son­able. They all give en­tirely differ­ent an­swers as well:

1: Go through life as sad as pos­si­ble.

2: Go through life pre­tend­ing to be sad to get more and then ac­tu­ally be happy later.

3: Only om­nipo­tence will make you truly happy. Any­thing else is an end­less hor­ror.

4: Life is pain, and the box is try­ing to sell you some­thing use­less, ig­nore it and move on.

• If chang­ing my util­ity func­tion has ex­pected pos­i­tive re­sults, based both on my cur­rent util­ity func­tion and in the pro­posed change, then…

Here the prob­lem is that the utilon is not a unit that can be con­verted into any other unit, in­clud­ing phys­i­cal phe­nom­ena.

• If you know the prob­a­bil­ity dis­tri­bu­tion P(t) of you dy­ing on day t, then you can solve ex­actly for op­ti­mal ex­pected life­time utilons out of the box. If you don’t know P(t), you can do some sort of adap­tive es­ti­ma­tion as you go.

• Note that the prob­lem is mak­ing the coun­ter­fac­tual as­sump­tion that

You are im­mor­tal.

• P(t) = 0.

• Why is this an in­ter­est­ing prob­lem?

• I’m not suffi­ciently fa­mil­iar with my own in­ter­nal crite­ria of in­ter­est­ing-ness to ex­plain to you why I find it in­ter­est­ing. Sorry you don’t as well.

• If you re­turn to a state you have already been at, you know you are go­ing to be wait­ing for­ever and lose and get noth­ing.

You seem to be as­sum­ing here that re­turn­ing to a state you have already been at is equiv­a­lent to loop­ing your be­hav­ior, so that once a Tur­ing ma­chine re-en­ters a pre­vi­ously in­stan­ti­ated state it can­not ex­hibit any novel be­hav­ior. But this isn’t true. A Tur­ing ma­chine can be­have differ­ently in the same state pro­vided the in­put it reads off its tape is differ­ent. The be­hav­ior must loop only if the the com­bi­na­tion of Tur­ing ma­chine state and tape con­figu­ra­tion re­curs. But this need never hap­pen as long as the tape is in­finite. If there were an in­finite amount of stuff in the world, even a finite mind might be able to lev­er­age it to, say, count to an ar­bi­trar­ily high num­ber.

Now you might ob­ject that it is not only minds that are finite, but also the world. There just isn’t an in­finite amount of stuff out there. But that same con­straint also rules out the pos­si­bil­ity of the util­ity box you de­scribe. I don’t see how one could squeeze ar­bi­trar­ily large amounts of util­ity into some finite quan­tity of mat­ter.

• You have given rea­sons why re­quiring bounded util­ity func­tions and dis­count­ing the fu­ture are not ad­e­quate re­sponses to the prob­lem if con­sid­ered in­di­vi­d­u­ally. But your ob­jec­tion to the bounded util­ity func­tion re­sponse as­sumes that fu­ture util­ity isn’t dis­counted, and your ob­jec­tion to the dis­count­ing re­sponse as­sumes that the util­ity func­tion is un­bounded. So what if we re­quire both that the util­ity func­tion must be bounded and that fu­ture util­ity must be dis­counted ex­po­nen­tially? Doesn’t that get around the para­dox?

I re­mem­ber read­ing a while ago about a para­dox where you start with \$1, and can trade that for a 50% chance of \$2.01, which you can trade for a 25% chance of \$4.03, which you can trade for a 12.5% chance of \$8.07, etc (can’t re­mem­ber where I read it).

The prob­lem state­ment isn’t pre­cisely the same as what you spec­ify here, but were you think­ing of the ven­er­a­ble St. Peters­burg para­dox?

• If your util­ity func­tion is bounded and you dis­count the fu­ture, then pick an amount of time af­ter now, ep­silon, such that the dis­count­ing by then is neg­ligible. Then imag­ine that the box dis­ap­pears if you don’t open it by then. at t = now + ep­silon * 2^-1, the utilons dou­ble. At 2^-2, they dou­ble again. etc.

But if your dis­count­ing is so great that you do not care about the fu­ture at all, I guess you’ve got me.

This isn’t the St. Peters­burg para­dox (though I al­most men­tioned it) be­cause in that, you make your de­ci­sion once at the be­gin­ning.

• If your util­ity func­tion is bounded and you dis­count the fu­ture, then pick an amount of time af­ter now, ep­silon, such that the dis­count­ing by then is neg­ligible. Then imag­ine that the box dis­ap­pears if you don’t open it by then. at t = now + ep­silon * 2^-1, the utilons dou­ble. At 2^-2, they dou­ble again. etc.

Per­haps I am mis­in­ter­pret­ing you, but I don’t see how this scheme is com­pat­i­ble with a bounded util­ity func­tion. For any bound n, there will be a time prior to ep­silon where the utilons in the box will be greater than n.

When you say “At 2^-2...”, I read that as “At now + ep­silon 2^-1 + ep­silon 2^-2...”. Is that what you meant?

• yeah, that’s what I meant. Also, in­stead of dou­bling, make it so they ex­po­nen­tially de­cay to­ward the bound.

• You could build a ma­chine that opens the box far in the fu­ture, at the mo­ment when the ma­chine’s re­li­a­bil­ity starts de­grad­ing faster than the utilons in­crease. This max­i­mizes your ex­pected util­ity.

Or if you’re not al­lowed to build a ma­chine, you sim­ply do the same with your­self (de­pend­ing on our model, pos­si­bly mul­ti­ply­ing by your ex­pected re­main­ing lifes­pan).

• Bring­ing to­gether what oth­ers have said, I pro­pose a solu­tion in three steps:

• Adopt a mixed strat­egy where, for each day, you open the box on that day with prob­a­bil­ity p. The ex­pected util­ity of this strat­egy is the sum of (p (1-p)^n 2^n), for n=0… which di­verges for any p in the half-open in­ter­val (0,0.5]. In other words, you get in­finite EU as long as p is in (0,0.5]. This is para­dox­i­cal, be­cause it means a strat­egy with a 0.5 risk of end­ing up with only 1 utilon is as good as any other.

• Ex­tend the range of our util­ity func­tion to a num­ber sys­tem with differ­ent in­fini­ties, where a faster-grow­ing se­ries has greater value than a slower-grow­ing se­ries, even if they both grow with­out bound. Now the EU of the mixed strat­egy con­tinues to grow as p ap­proaches 0, bring­ing us back to the origi­nal prob­lem: The smaller p is, the bet­ter, but there is no small­est pos­i­tive real num­ber.

• Real­ize that phys­i­cal agents can only choose be­tween a finite num­ber of strate­gies (be­cause we only have a finite num­ber of pos­si­ble mind states). So, in prac­tice, there is always a small­est p: the small­est p we can im­ple­ment in re­al­ity.

So that’s it. Build a ran­dom num­ber gen­er­a­tor with as many bits of pre­ci­sion as pos­si­ble. Run it ev­ery day un­til it out­puts 0. Then open the box. This strat­egy im­proves on the OP be­cause it yields in­finite ex­pected pay­out, and is in­tu­itively ap­peal­ing be­cause it also has a very high me­dian pay­out, with a very small prob­a­bil­ity of a low pay­out. Also, it doesn’t re­quire pre­com­mit­ment, which seems more math­e­mat­i­cally el­e­gant be­cause it’s a time-sym­met­ric strat­egy for a time-sym­met­ric game.

• There is a good rea­son to use a bounded util­ity func­tion, note—if the con­di­tions of Sav­age’s the­o­rem hold, the util­ity func­tion you get from it is bounded.

• How long do you wait be­fore open­ing it? If you never open it, you get noth­ing (you lose! Good day, sir or madam!) and when­ever you take it, tak­ing it one day later would have been twice as good.

When do I “lose” pre­cisely? When I never take it? By happy co­in­ci­dence ‘never’ hap­pens to be the very next day af­ter I planned to open the box!

• There are no other ways to get utilons.

Is a weak­ness in your ar­gu­ment. Either you can sur­vive with­out utilons, a con­tra­dic­tion to util­ity the­ory, or you wait un­til your “pre-ex­ist­ing” utilons are used up and you need more to sur­vive.

• This sug­gests a joke solu­tion: Tell peo­ple about the box, then ask them for a loan which you will re­pay with pro­ceeds from the box. Then you can live off the loan and let your cred­i­tors worry about solv­ing the un­solv­able.

• Is a weak­ness in your ar­gu­ment. Either you can sur­vive with­out utilons, a con­tra­dic­tion to util­ity the­ory, or you wait un­til your “pre-ex­ist­ing” utilons are used up and you need more to sur­vive.

Utilons don’t need to be as­so­ci­ated with sur­vival. Sur­vival can be a mere in­stru­men­tal good used to in­crease the amount of ac­tual utilons gen­er­ated (by mak­ing, say, pa­per­clips). I get the im­pres­sion that you mean some­thing differ­ent by the word than what the post (and the site) mean.

• What’s wrong with not hav­ing any more rea­son to live af­ter you get the utilons?

• I see you found yet an­other prob­lem, with no way to get more utilons you die when those in the box are used up. And util­ity the­ory says you need util­ity to live, not just to give you rea­son to live.

• And util­ity the­ory says you need util­ity to live, not just to give you rea­son to live.

This is con­trary to my un­der­stand­ing. Can you ex­plain, please?

• If I am ac­tu­ally im­mor­tal, and there is no other way to get Utilions then each day, the value of me open­ing the box is some­thing like:

Value=Utilions/​Fu­ture Days

Since my Fu­ture Days are sup­pos­edly in­finite, we are talk­ing about at best an in­finites­i­mal differ­ence be­tween me open­ing the box on Day 1 and me open­ing the box on Day 3^^^^3. There is no ac­tual wrong day to open the box. If that seems im­plau­si­ble, it is be­cause the hy­po­thet­i­cal it­self is im­plau­si­ble.

• If I am ac­tu­ally im­mor­tal, and there is no other way to get Utilions then each day, the value of me open­ing the box is some­thing like:

Value=Utilions/​Fu­ture Days

The ex­pected value of open­ing the box is:

Value=Utilons

That is all. That num­ber already rep­re­sents how much value is as­signed to the state of the uni­verse given that de­ci­sion. Di­vid­ing by only fu­ture days is an er­ror. As­sign­ing a differ­ent value to the speci­fied re­ward based on whether days are in the past or the fu­ture changes the prob­lem.

• Pre­sum­ably, if Utilions are use­ful at all, then you use them. Usu­ally, this means that some are lost each day in the pro­cess of us­ing them.

Fur­ther, un­less the Utilions rep­re­sent some re­source that is non-en­tropic, then I will lose some num­ber of Utilions each day even if they aren’t lost by me us­ing them. This works out to the same an­swer in the long run.

Let’s as­sume we have an agent Boxxy, an im­mor­tal AI whose util­ity func­tion is that open­ing the box to­mor­row is twice as good as open­ing it to­day. Once he opens the box, his util­ity func­tion as­signs that much value to the uni­verse. Let’s as­sume this is all he val­ues. (This gets us around a num­ber of prob­lems for the sce­nario.)

Even in this sce­nario, un­less Boxxy is im­mune to en­tropy, some amount of in­for­ma­tion (and thus, some per­cep­tion of util­ity) will be lost over time. Over a long enough time, Boxxy will even­tu­ally lose the mem­ory of open­ing the Box. Even if Boxxy is ca­pa­ble of self-re­pair in the face of en­tropy, un­less Boxxy is ca­pa­ble of ac­tu­ally not un­der­go­ing en­tropy, some of the Box-in­for­ma­tion will be lost. (Maybe Boxxy hopes that it can re­place it with an iden­ti­cal mem­ory for its util­ity func­tion, al­though I would sus­pect at that point Boxxy might just to de­cide to re­mem­ber hav­ing opened the Box at a nearer fu­ture date) Even­tu­ally, Boxxy’s mem­ory and thus, Boxxy’s Utilions, will ei­ther be com­pletely ar­tifii­cal with at best some­thing like a causal re­la­tion­ship to pre­vi­ous mem­ory states of open­ing the box, or Boxxy will lose all of its Utilions.

Of course, Boxxy might never open the box. (I am not a su­per­in­tel­li­gence ob­sessed with box open­ing. I am a hu­man in­tel­li­gence ob­sessed with things that Boxxy would find ir­rele­vant. So I can only guess as to what a box-based AGI would do.) In this case, the Utilions won’t de­grade, but Boxxy can still ex­pect a value of 0 in this case.

Frankly, the prob­lem is hard to think about at that level, be­cause real im­mor­tal­ity (as the prob­lem re­quires) would re­quire some­way to en­sure that en­tropy doesn’t oc­cur but some­how some sort of pro­cess oc­curs, which seems a con­tra­dic­tion in terms. I guess this could be oc­cur­ing in a uni­verse with­out en­tropy, (but which some­how has other pro­cesses) al­though both my in­tu­itions and my knowl­edge are so firmly rooted in a uni­verse that has en­tropy that I don’t have a good ground­ing on how to eval­u­ate prob­lems in such a uni­verse.

• Pre­sum­ably, if Utilions are use­ful at all, then you use them. Usu­ally, this means that some are lost each day in the pro­cess of us­ing them.

No. Those are re­sources.

• I ad­mire the way this post in­tro­duces an in­ge­nious prob­lem and an in­ge­nious an­swer.