# The Number Choosing Game: Against the existence of perfect theoretical rationality

In or­der to en­sure that this post de­liv­ers what it promises, I have added the fol­low­ing con­tent warn­ings:

Con­tent Notes:

Pure Hy­po­thet­i­cal Si­tu­a­tion
: The claim that perfect the­o­ret­i­cal ra­tio­nal­ity doesn’t ex­ist is re­stricted to a purely hy­po­thet­i­cal situ­a­tion. No claim is be­ing made that this ap­plies to the real world. If you are only in­ter­ested in how things ap­ply to the real world, then you may be dis­ap­pointed to find out that this is an ex­er­cise left to the reader.

Tech­ni­cal­ity Only Post: This post ar­gues that perfectly the­o­ret­i­cal ra­tio­nal­ity doesn’t ex­ist due to a tech­ni­cal­ity. If you were hop­ing for this post to de­liver more, well, you’ll prob­a­bly be dis­ap­pointed.

Con­tentious Defi­ni­tion: This post (roughly) defines perfect ra­tio­nal­ity as the abil­ity to max­imise util­ity. This is based on Wikipe­dia, which defines ra­tio­nal agents as an agent that: “always chooses to perform the ac­tion with the op­ti­mal ex­pected out­come for it­self from among all fea­si­ble ac­tions”.

We will define the num­ber choos­ing game as fol­lows. You name any sin­gle finite num­ber x. You then gain x util­ity and the game then ends. You can only name a finite num­ber, nam­ing in­finity is not al­lowed.

Clearly, the agent that names x+1 is more ra­tio­nal than the agent that names x (and be­haves the same in ev­ery other situ­a­tion). How­ever, there does not ex­ist a com­pletely ra­tio­nal agent, be­cause there does not ex­ist a num­ber that is higher than ev­ery other num­ber. In­stead, the agent who picks 1 is less ra­tio­nal than the agent who picks 2 who is less ra­tio­nal than the agent who picks 3 and so on un­til in­finity. There ex­ists an in­finite se­ries of in­creas­ingly ra­tio­nal agents, but no agent who is perfectly ra­tio­nal within this sce­nario.

Fur­ther­more, this hy­po­thet­i­cal doesn’t take place in our uni­verse, but in a hy­po­thet­i­cal uni­verse where we are all ce­les­tial be­ings with the abil­ity to choose any num­ber how­ever large with­out any ad­di­tional time or effort no mat­ter how long it would take a hu­man to say that num­ber. Since this state­ment doesn’t ap­pear to have been clear enough (judg­ing from the com­ments), we are ex­plic­itly con­sid­er­ing a the­o­ret­i­cal sce­nario and no claims are be­ing made about how this might or might not carry over to the real world. In other words, I am claiming the the ex­is­tence of perfect ra­tio­nal­ity does not fol­low purely from the laws of logic. If you are go­ing to be difficult and ar­gue that this isn’t pos­si­ble and that even hy­po­thet­i­cal be­ings can only com­mu­ni­cate a finite amount of in­for­ma­tion, we can imag­ine that there is a de­vice that pro­vides you with util­ity the longer that you speak and that the util­ity it pro­vides you is ex­actly equal to the util­ity you lose by hav­ing to go to the effort to speak, so that over­all you are in­differ­ent to the re­quired speak­ing time.

In the com­ments, MattG sug­gested that the is­sue was that this prob­lem as­sumed un­bounded util­ity. That’s not quite the prob­lem. In­stead, we can imag­ine that you can name any num­ber less than 100, but not 100 it­self. Fur­ther, as above, say­ing a long num­ber ei­ther doesn’t cost you util­ity or you are com­pen­sated for it. Re­gard­less of whether you name 99 or 99.9 or 99.9999999, you are still choos­ing a sub­op­ti­mal de­ci­sion. But if you never stop speak­ing, you don’t re­ceive any util­ity at all.

I’ll ad­mit that in our uni­verse there is a perfectly ra­tio­nal op­tion which bal­ances speak­ing time against the util­ity you gain given that we only have a finite life­time and that you want to try to avoid dy­ing in the mid­dle of speak­ing the num­ber which would re­sult in no util­ity gained. How­ever, it is still no­table that a perfectly ra­tio­nal be­ing can­not ex­ist within a hy­po­thet­i­cal uni­verse. How ex­actly this re­sult ap­plies to our uni­verse isn’t ex­actly clear, but that’s the challenge I’ll set for the com­ments. Are there any re­al­is­tic sce­nar­ios where the lack of ex­is­tence of perfect ra­tio­nal­ity has im­por­tant prac­ti­cal ap­pli­ca­tions?

Fur­ther­more, there isn’t an ob­jec­tive line be­tween ra­tio­nal and ir­ra­tional. You or I might con­sider some­one who chose the num­ber 2 to be stupid. Why not at least go for a mil­lion or a billion? But, such a per­son could have eas­ily gained a billion, billion, billion util­ity. No mat­ter how high a num­ber they choose, they could have always gained much, much more with­out any differ­ence in effort.

I’ll finish by pro­vid­ing some ex­am­ples of other games. I’ll call the first game the Ex­plod­ing Ex­po­nen­tial Coin Game. We can imag­ine a game where you can choose to flip a coin any num­ber of times. Ini­tially you have 100 util­ity. Every time it comes up heads, your util­ity triples, but if it comes up tails, you lose all your util­ity. Fur­ther­more, let’s as­sume that this agent isn’t go­ing to raise the Pas­cal’s Mug­ging ob­jec­tion. We can see that the agent’s ex­pected util­ity will in­crease the more times they flip the coin, but if they com­mit to flip­ping it un­limited times, they can’t pos­si­bly gain any util­ity. Just as be­fore, they have to pick a finite num­ber of times to flip the coin, but again there is no ob­jec­tive jus­tifi­ca­tion for stop­ping at any par­tic­u­lar point.

Another ex­am­ple, I’ll call the Un­limited Swap game. At the start, one agent has an item worth 1 util­ity and an­other has an item worth 2 util­ity. At each step, the agent with the item worth 1 util­ity can choose to ac­cept the situ­a­tion and end the game or can swap items with the other player. If they choose to swap, then the player who now has the 1 util­ity item has an op­por­tu­nity to make the same choice. In this game, wait­ing for­ever is ac­tu­ally an op­tion. If your op­po­nents all have finite pa­tience, then this is the best op­tion. How­ever, there is a chance that your op­po­nent has in­finite pa­tience too. In this case you’ll both miss out on the 1 util­ity as you will wait for­ever. I sus­pect that an agent could do well by hav­ing a chance of wait­ing for­ever, but also a chance of stop­ping af­ter a high finite num­ber. In­creas­ing this finite num­ber will always make you do bet­ter, but again, there is no max­i­mum wait­ing time.

(This seems like such an ob­vi­ous re­sult, I imag­ine that there’s ex­ten­sive dis­cus­sion of it within the game the­ory liter­a­ture some­where. If any­one has a good pa­per that would be ap­pre­ci­ated).

Link to part 2: Con­se­quences of the Non-Ex­is­tence of Ra­tion­al­ity

• There is some con­fu­sion in the com­ments over what util­ity is.

the max­i­mum util­ity that it could con­ceiv­ably ex­pect to use

and Usul writes:

goes out to spend his util­ity on black­jack and hookers

Utility is not a re­source. It is not some­thing that you can ac­quire and then use, or save up and then spend. It is not that sort of thing. It is noth­ing more than a nu­mer­i­cal mea­sure of the value you as­cribe to some out­come or state of af­fairs. The black­jack and hook­ers, if that’s what you’re into, are the things that you would be speci­fi­cally seek­ing by seek­ing the high­est util­ity, not some­thing you would af­ter­wards get in ex­change for some ac­quired quan­tity of util­ity.

• Heat Death still comes into play. If you stand there calcu­lat­ing high num­bers for longer than that, or mash­ing on the 9 key, or swap­ping 1 utilon for 2 (or 2 billion), it never mat­ters. You still end up with zero at the end of things.

ETA: If you come back an tell me that “these sce­nar­ios as­sume an un­limited availa­bil­ity of time” or some­thing like that, I’ll ask to see if the dragon in your garage is per­me­able to flour.

• Note that I am not the per­son mak­ing the ar­gu­ment, just clar­ify­ing what is meant by “util­ity”, which in its use around here speci­fi­cally means that which is con­structed by the VNM the­o­rem. I am not a par­tic­u­lar fan of ap­ply­ing the con­cept to uni­ver­sal de­ci­sion-mak­ing.

You still end up with zero at the end of things.

Are you ar­gu­ing that all things end, there­fore there is no value in any­thing?

Well, there is prece­dent:

All is van­ity. What does man gain by all the toil at which he toils un­der the sun?

I said in my heart, “Come now, I will test you with plea­sure; en­joy your­self.” But be­hold, this also was van­ity. I said of laugh­ter, “It is mad,” and of plea­sure, “What use is it?”

Then I con­sid­ered all that my hands had done and the toil I had ex­pended in do­ing it, and be­hold, all was van­ity and a striv­ing af­ter wind, and there was noth­ing to be gained un­der the sun.

The wise per­son has his eyes in his head, but the fool walks in dark­ness. And yet I per­ceived that the same event hap­pens to all of them. Then I said in my heart, “What hap­pens to the fool will hap­pen to me also. Why then have I been so very wise?” And I said in my heart that this also is van­ity. For of the wise as of the fool there is no en­dur­ing re­mem­brance, see­ing that in the days to come all will have been long for­got­ten. How the wise dies just like the fool! So I hated life, be­cause what is done un­der the sun was grievous to me, for all is van­ity and a striv­ing af­ter wind.

• re­views VNM Theorem

Noted, and thanks for the up­date. :)

• Are you ar­gu­ing that all things end, there­fore there is no value in any­thing?

My ar­gu­ment was not meant to im­ply nihilism, though that is an in­ter­est­ing point. (Aside: Where is the quote from?) Rather, I meant to im­ply the hid­den costs (e.g. time for calcu­la­tion or in­put) mak­ing the ex­er­cise mean­ingless. As has been ar­gued by sev­eral peo­ple now, hav­ing the Agent be able to state ar­bi­trar­ily large or ac­cu­rate num­bers, or able to wait an ar­bi­trar­ily large amount of time with­out los­ing any util­ity is… let’s say prob­le­matic. As much so as the likely­hood of the Game Master be­ing able to ac­tu­ally hand out util­ity based on an ar­bi­trar­ily large/​ac­cu­rate num­ber.

• The quo­ta­tion is from the bibli­cal Book of Ec­cle­si­astes, tra­di­tion­ally (but prob­a­bly wrongly) as­cribed to the allegedly very wise King Solomon.

• Heat death is a prob­lem that the builders of the game have to deal with. Every time I type out BB(BB(BB(...))) the builder of the game has to figure out how I can get a non­com­putable in­crease to the de­gree of the func­tion by which the mul­ti­ple of my prefer­ence for the world in­creases. If there is some con­ceiv­able world with no heat death which I pre­fer any com­putable amount more than any world with a heat death (and in­finity is not a util­ity!), then by play­ing this game I en­ter such a world.

• Not if your cur­rent uni­verse ends be­fore you are able to finish spec­i­fy­ing the num­ber. Re­mem­ber: you re­ceive no util­ity be­fore you com­plete your in­put.

• “If you come back an tell me that “these sce­nar­ios as­sume an un­limited availa­bil­ity of time” or some­thing like that, I’ll ask to see if the dragon in your garage is per­me­able to flour.”

Not be­ing re­al­is­tic is not a valid crit­i­cism of a the­o­ret­i­cal situ­a­tion if the the­o­ret­i­cal situ­a­tion is not meant to rep­re­sent re­al­ity. I’ve made no claims of how it car­ries over to the real world

• “Not re­al­is­tic” isn’t my ob­jec­tion here so much as “mov­ing the goal­post”. The origi­nal post (as I re­call it from be­fore the edit), made no claim that there was zero cost in spec­i­fy­ing ar­bi­trar­ily large/​spe­cific num­bers, nor in par­ti­ci­pat­ing in ar­bi­trar­ily large num­bers of swaps.

• It’s been like that from the start. EDIT: I only added in ex­tra clar­ifi­ca­tion.

• I cer­tainly make no claims about the perfect qual­ity of my mem­ory. ;)

• So, when we solve lin­ear pro­gram­ming prob­lems (say, with the sim­plex method), there are three pos­si­ble out­comes: the prob­lem is in­fea­si­ble (there are no solu­tions that satisfy the con­straints), the prob­lem has at least one op­ti­mal value (which is found), or the prob­lem is un­bounded.

That is, if your “perfect the­o­ret­i­cal ra­tio­nal­ity” re­quires there to not be the pos­si­bil­ity of un­bounded solu­tions, then your perfect the­o­ret­i­cal ra­tio­nal­ity won’t work and can­not in­clude sim­ple things like LP prob­lems. So I’m not sure why you think this ver­sion of perfect the­o­ret­i­cal ra­tio­nal­ity is in­ter­est­ing, and am mildly sur­prised and dis­ap­pointed that this was your im­pres­sion of ra­tio­nal­ity.

• “Can­not in­clude sim­ple things like LP prob­lems”—Well, lin­ear pro­gram­ming prob­lems are sim­ply a more com­plex ver­sion of the num­ber choos­ing game. In fact, the num­ber choos­ing game is equiv­a­lent to lin­ear pro­gram­ming max­imis­ing x with x>0. So, if you want to crit­i­cise my defi­ni­tion of ra­tio­nal­ity for not be­ing able to solve ba­sic prob­lems, you should be crit­i­cis­ing it for not be­ing able to solve the num­ber choos­ing game!

I wouldn’t say it makes this un­in­ter­est­ing though, as while it may seem ob­vi­ous to you that perfect ra­tio­nal­ity as defined by util­ity max­imi­sa­tion is im­pos­si­ble, as you have ex­pe­rience with lin­ear pro­gram­ming, it isn’t nec­es­sar­ily ob­vi­ous to ev­ery­one else. In fact, if you read the com­ments, you’ll see that many com­men­ta­tors are un­will­ing to ac­cept this solu­tion and keep try­ing to in­sist on there be­ing some way out.

You seem to be ar­gu­ing that there must be some solu­tion that can solve these prob­lems. I’ve already proven that this can­not ex­ist, but if you dis­agree, what is your solu­tion then?

EDIT: Essen­tially, what you’ve done is take some­thing “ab­surd” (that there is no perfect ra­tio­nal­ity for the num­ber choos­ing game), re­duce it to some­thing less ab­surd (that there’s no perfect ra­tio­nal­ity for lin­ear pro­gram­ming) and then de­clared that you’ve found a re­duc­tio ad ab­sur­dum. That’s not how it is sup­posed to work!

• You seem to be ar­gu­ing that there must be some solu­tion that can solve these prob­lems. I’ve already proven that this can­not ex­ist, but if you dis­agree, what is your solu­tion then?

I think you’re mi­s­un­der­stand­ing me. I’m say­ing that there are prob­lems where the right ac­tion is to mark it “un­solv­able, be­cause of X” and then move on. (Here, it’s “un­solv­able be­cause of un­bounded solu­tion space in the in­creas­ing di­rec­tion,” which is true in both the “pick a big num­ber” and “open bound­ary at 100″ case.)

In fact, if you read the com­ments, you’ll see that many com­men­ta­tors are un­will­ing to ac­cept this solu­tion and keep try­ing to in­sist on there be­ing some way out.

Sure, some­one who is ob­ject­ing that this prob­lem is ‘solv­able’ is not us­ing ‘solv­able’ the way I would. But some­one who is ob­ject­ing that this prob­lem is ‘un­fair’ be­cause it’s ‘im­pos­si­ble’ is start­ing down the cor­rect path.

then de­clared that you’ve found a re­duc­tio ad ab­sur­dum.

I think you have this in re­verse. I’m say­ing “the re­sult you think is ab­surd is nor­mal in the gen­eral case, and so is nor­mal in this spe­cial case.”

• I think you’re mi­s­un­der­stand­ing me. I’m say­ing that there are prob­lems where the right ac­tion is to mark it “un­solv­able, be­cause of X” and then move on. (Here, it’s “un­solv­able be­cause of un­bounded solu­tion space in the in­creas­ing di­rec­tion,” which is true in both the “pick a big num­ber” and “open bound­ary at 100″ case.)

But if we view this as an ac­tual (albeit un­re­al­is­tic/​highly the­o­ret­i­cal) situ­a­tion rather than a math prob­lem we are still stuck with the ques­tion of which ac­tion to take. A perfectly ra­tio­nal agent can re­al­ize that the prob­lem has no op­ti­mal solu­tion and mark it as un­solv­able, but af­ter­wards they still have to pick a num­ber, so which num­ber should they pick?

• But if we view this as an ac­tual (albeit un­re­al­is­tic/​highly the­o­ret­i­cal) situation

There is no such thing as an ac­tual un­re­al­is­tic situ­a­tion.

A perfectly ra­tio­nal agent can re­al­ize that the prob­lem has no op­ti­mal solu­tion and mark it as un­solv­able, but af­ter­wards they still have to pick a number

They do not have to pick a num­ber, be­cause the situ­a­tion is not real. To say “but sup­pose it was” is only to re­peat the origi­nal hy­po­thet­i­cal ques­tion that the agent has de­clared un­solved. If we stipu­late that the agent is so log­i­cally om­ni­scient as to never need to aban­don a prob­lem as un­solved, that does not tell us, who are not om­ni­scient, what that hy­po­thet­i­cal agent’s hy­po­thet­i­cal choice in that hy­po­thet­i­cal situ­a­tion would be.

The whole prob­lem seems to me on a level with “can God make a weight so heavy he can’t lift it?”

• UPDATED: If asked whether the prob­lem is solv­able, a perfectly ra­tio­nal agent would re­ply that it isn’t.

If asked what ac­tion to take, then the perfectly ra­tio­nal agent is stuck, and there­fore finds out it isn’t perfect. Those are two dis­tinct ques­tions. I sup­pose it all comes down to how you define ra­tio­nal­ity though.

• So, be­sides the is­sue of what I will call ear­lier work, CCC and oth­ers have already men­tioned that your sce­nario would al­low non-con­verg­ing ex­pected val­ues as in the St Peters­burg para­dox. By the usual mean­ing of util­ity, which you’ll note is not ar­bi­trary but equiv­a­lent to cer­tain at­trac­tive ax­ioms, your sce­nario con­tra­dicts it­self.

I’ve seen two main solu­tions offered for the gen­eral prob­lem. If we just re­quire bounded util­ity, there might be some­thing left of the OP—but only with as­sump­tions that ap­pear phys­i­cally im­pos­si­ble and thus highly sus­pi­cious. (Im­me­di­ately af­ter learn­ing your ar­gu­ment con­tra­dicted it­self is a bad time to tell us what you think is log­i­cally pos­si­ble!) I tend to­wards the other op­tion, which says the peo­ple com­plain­ing about physics are onto some­thing fun­da­men­tal con­cern­ing the prob­a­bil­ities of ever-vaster util­ities. This would dis­in­te­grate the OP en­tirely.

• “Non-con­verg­ing ex­pected val­ues”—you can’t con­clude that the sce­nario is con­tra­dic­tory just be­cause your tools don’t work.

As already noted, we can con­sider the prob­lem where you name any num­ber less than 100, but not 100 it­self and gain that much util­ity, which avoids the whole non-con­ver­gence prob­lem.

“This would dis­in­te­grate the OP en­tirely”—as already stated in other com­ments, claims that my situ­a­tion aren’t re­al­is­tic would be a good crit­i­cism if I was claiming that the re­sults could be di­rectly ap­plied to the real uni­verse.

• If asked whether the prob­lem is solv­able, a perfectly ra­tio­nal agent would re­ply that it is.

Why? It’s a prob­lem with­out a solu­tion. Would a perfect ra­tio­nal agent say the prob­lem of find­ing a nega­tive in­te­ger that’s greater than 2 is solv­able?

• Sorry, that was a typo. It was meant to say “isn’t” rather than “is”

• The is­sue here isn’t that ra­tio­nal­ity is im­pos­si­ble. The is­sue here is that you’re let­ting an un­defined ab­stract con­cept do all your heavy lift­ing, and tak­ing it places it can­not mean­ingfully be.

Utili­tar­i­anism: Defin­ing “Good” is hard. Math is easy, let X stand in for “Good”, and we’ll max­i­mize X, thereby max­i­miz­ing “Good”.

So let’s do some sub­sti­tu­tion. Let’s say ap­ples are good. Would you wait for­ever for an ap­ple? No? What if make it so you live for­ever? No, you’d get bored? What if we make it so that you don’t get bored wait­ing? No, you have other things that have more value to you? Well, we’ll put you in a (sci­ence fic­tion words) closed time loop, so that no mat­ter how long you spend trad­ing the ap­ple back and forth, you’ll come out with­out hav­ing lost any­thing? And so on and so forth, un­til all the countless po­ten­tial ob­jec­tions are elimi­nated.

Keep go­ing un­til all that’s left is one ex­tra ap­ple, and the ra­tio­nal thing to do is to wait for­ever for an ap­ple you’ll never end up with. One by one, you’ve elimi­nated ev­ery rea­son -not- to wait for­ever—why should it sur­prise you that wait­ing for­ever is the cor­rect thing to do, when you’ve gone to some much trou­ble to make sure that it is the cor­rect thing to do?

Your “What’s the high­est num­ber game” is, well, a “What’s the high­est num­ber game”. Let’s put this in con­crete terms: Who­ever names the high­est num­ber gets \$1,000. There are now two var­i­ants of the game: In the first var­i­ant, you get an in­finite num­ber of turns. I think it’s ob­vi­ous this is iden­ti­cal to the Ap­ple swap­ping game. In the sec­ond var­i­ant, you get ex­actly one turn to name a num­ber. Ap­ply all the con­straints of the Ap­ple swap­ping game, such that there is no cost to the player for tak­ing longer. Well, the ob­vi­ous strat­egy now is to keep re­peat­ing the num­ber “9” un­til you’ve said it more times than your op­po­nent. And we’re back to the ap­ple swap­ping game. There’s no cost to con­tin­u­ing.

What makes all this seem to break ra­tio­nal­ity? Be­cause we don’t live in a uni­verse with­out costs, and our brains are hard­wired to con­sider costs. If you find your­self in a uni­verse with­out costs, where you can ob­tain an in­finite amount of util­ity by re­peat­ing the num­ber “9” for­ever, well, keep re­peat­ing the num­ber “9″ for­ever, along with ev­ery­body else in the uni­verse. It’s not like you’ll ever get bored or have some­thing more im­por­tant to do.

• “Keep go­ing un­til all that’s left is one ex­tra ap­ple, and the ra­tio­nal thing to do is to wait for­ever for an ap­ple you’ll never end up with”—that doesn’t re­ally fol­low. You have to get the Ap­ple and exit the time loop at some point or you never get any­thing.

“If you find your­self in a uni­verse with­out costs, where you can ob­tain an in­finite amount of util­ity by re­peat­ing the num­ber “9” for­ever, well, keep re­peat­ing the num­ber “9″ for­ever, along with ev­ery­body else in the uni­verse.”—the sce­nario speci­fi­cally re­quires you to ter­mi­nate in or­der to gain any util­ity.

• But ap­par­ently you are not los­ing util­ity over time? And hold­ing util­ity over time isn’t of value to me, oth­er­wise my failure to ter­mi­nate early is cost­ing me the util­ity I didn’t take at that point in time? If there’s a lever com­pen­sat­ing for that loss of util­ity then I’m ac­tu­ally gain­ing the util­ity I’m turn­ing down any­way!

Ba­si­cally the only rea­son to stop at time t1 would be that you will re­gret not hav­ing had the util­ity available at t1 un­til t2, when you de­cide to stop.

• “Ba­si­cally the only rea­son to stop at time t1 would be that you will re­gret not hav­ing had the util­ity available at t1 un­til t2, when you de­cide to stop.”—In this sce­nario, you re­ceive the util­ity when you stop speak­ing. You can speak for an ar­bi­trar­ily long amount of time and it doesn’t cost you any util­ity as you are com­pen­sated for any util­ity that it would cost, but if you never stop speak­ing you never gain any util­ity.

• Then the “ra­tio­nal” thing is to never stop speak­ing. It’s true that by never stop­ping speak­ing I’ll never gain util­ity but by stop­ping speak­ing early I miss out on fu­ture util­ity.

The be­havi­our of speak­ing for­ever seems ir­ra­tional, but you have de­liber­ately crafted a sce­nario where my only goal is to get the high­est pos­si­ble util­ity, and the only way to do that is to just keep speak­ing. If you sug­gest that some­one who got some util­ity af­ter 1 mil­lion years is “more ra­tio­nal” than some­one still speak­ing at 1 billion years then you are adding a value judg­ment not ap­par­ent in the origi­nal sce­nario.

• In­finite util­ity is not a pos­si­ble util­ity in the sce­nario and there­fore the be­havi­our of not stop­ping is not a high­est pos­si­ble util­ity. Con­tinue to speak is an im­prove­ment only given that you do stop at some time. If you con­tinue by not stop­ping ever you get 0 util­ity which is lower than speak­ing a 2 digit num­ber.

• But time doesn’t end. The crite­ria of as­sess­ment is

1)I only care about get­ting the high­est num­ber possible

2)I am ut­terly in­differ­ent to how long this takes me

3)The only way to gen­er­ate this value is by speak­ing this num­ber (or, at the very least, any other meth­ods I might have used in­stead are com­pen­sated ex­plic­itly once I finish speak­ing).

If your ar­gu­ment is that Bob, who stopped at Gra­hams num­ber, is more ra­tio­nal than Jim, who is still speak­ing, then you’ve changed the terms. If my goal is to beat Bob, then I just need to stop at Gra­ham’s num­ber plus one.

At any given time, t, I have no rea­son to stop, be­cause I can ex­pect to earn more by con­tin­u­ing. The only rea­son this looks ir­ra­tional is we are imag­in­ing things which the sce­nario rules out: time costs or in­finite time com­ing to an end.

The ar­gu­ment “but then you never get any util­ity” is true, but that doesn’t mat­ter, be­cause I last for­ever. There is no end of time in this sce­nario.

If your ar­gu­ment is that in a uni­verse with in­finite time, in­finite life and a magic in­cen­tive but­ton then all ev­ery­one will do is press that but­ton for­ever then you are cor­rect, but I don’t think you’re say­ing much.

• python code of

while True: pass oh­noes=1/​0

doesn’t gen­er­ate a run­time ex­cep­tion when ran

similiarly

util­ity=0 a=0 while True: a+=1 util­ity+=a

doesn’t as­sign to util­ity more than once

in contrast

util­ity=0 while True: util­ity+=1

does as­sign to util­ity more than once. With finite iter­a­tions these two would be quite in­ter­change­able but with non-ter­mi­nat­ing iter­a­tions its not. The iter­a­tion doesn’t need to ter­mi­nate for this to be true.

Say you are in a mar­ket and you know some­one who sells wheat for \$5 and some­one who buys it for \$10 and some­one who sells wine for \$7 and sup­pose that you care about wine. If you have a strat­egy that only con­sists of buy­ing and sel­l­ing wheat you don’t get any wine. There needs to be a “cashout” move of buy­ing wine atleast once. Now think of a situ­a­tion that when you buy wine you need to hand over your wheat deal­ing li­cence. Well a wheat li­cence means ar­bi­trary amounts of wine so ir­ra­tional to ever trade wheat li­cense away for a finite amount of wine right? But then you end up with a wine “max­imis­ing strat­egy” that does so by not ever buy­ing wine.

• In­deed. And that’s what hap­pens when you give a max­imiser per­verse in­cen­tives and in­finity in which to gain them.

This sce­nario cor­re­sponds pre­cisely to pseu­docode of the kind

new­val<-1

old­val<-0

while new­val>oldval

{

old­val<-newval

new­val<-new­val+1

}

Which never ter­mi­nates. This is only ir­ra­tional if you want to ter­mi­nate (which you usu­ally do), but again, the claim that the max­imiser never ob­tains value doesn’t mat­ter be­cause you are es­sen­tially plac­ing an out­side judg­ment on the sys­tem.

Ba­si­cally, what I be­lieve you (and the op) are do­ing is look­ing at two agents in the num­ber­verse.

Agent one stops at time 100 and gains X util­ity Agent two con­tinues for­ever and never gains any util­ity.

Clearly, you think, agent one has “won”. But how? Agent two has never failed. The num­ber­verse is eter­nal, so there is no point at which you can say it has “lost” to agent one. If the num­ber­verse had a non zero prob­a­bil­ity of col­laps­ing at any point in time then Agent two’s strat­egy would in­stead be more com­plex (and pos­si­bly un­com­putable if we dis­tribute over in­finity), but as we are told that agent one and two ex­ist in a change­less uni­verse and their only goal is to ob­tain the most util­ity then we can’t judge ei­ther to have won. In fact agent two’s strat­egy only pre­vents it from los­ing, and it can’t win.

That is, if we imag­ine the num­ber­verse full of agents, any agent which chooses to stop will lose in a con­test of util­ity, be­cause the re­main­ing agents can always choose to stop and ob­tain their far greater util­ity. So the ra­tio­nal thing to do in this con­test is to never stop.

Sure, that’s a pretty bleak look­out, but as I say, if you make a situ­a­tion ar­tifi­cial enough you get ar­tifi­cial out­comes.

• What you are say­ing would be op­ti­mis­ing in a uni­verse where the agent gets the util­ity as it says the num­ber. Then the av­er­age util­ity of a un­goer would be greater than that of a idler.

How­ever if the util­ity is dished out af­ter the num­ber has been spe­sified then an idler and a on­goer have ex­actly the same amount of util­ity and ought to be as op­ti­mal. 0 is not a op­ti­mum of this game so an agent that re­sults in 0 util­ity is not an op­ti­miser. If you take an agent that is an op­ti­miser in other con­text then it of­course might not be an op­ti­miser for this game.

There is also the prob­lem that choos­ing the con­tinue doesn’t yield the utilty with cer­tainty only “al­most always”. The on­goer strat­egy hits pre­ci­cely in the hole in this cer­tainty when no pay­out hap­pens. I guess you may be able to define a game where con­cur­rently with their ac­tions. But this reeks of “the house” hav­ing pre­mo­ni­tion on what the agent is go­ing to do in­stead of in­fer­ring its from its ac­tions. if the rules are “first ac­tions and THEN pay­out” you need to be able to do your ac­tion to get a pay­out.

In the on­go­ing ver­sion I could think of rules that an agent that has said “9.9999...” to 400 digits would re­ceive 0.000.(401 ze­roes)..9 util­ity on the next digit. How­ever if the agents get util­ity as­signed only once there won’t be a “stand­ing so far”. How­ever this be­havi­our would then be the perfectly ra­tio­nal thing to do as there would be a uniquely de­ter­mined digit to keep on say­ing. I am sus­pect­ing the trou­ble is mix­ing the on­go­ing and the dis­patch ver­sion to each other in­con­sis­tently.

• “How­ever if the util­ity is dished out af­ter the num­ber has been spe­sified then an idler and a on­goer have ex­actly the same amount of util­ity and ought to be as op­ti­mal. 0 is not a op­ti­mum of this game so an agent that re­sults in 0 util­ity is not an op­ti­miser. If you take an agent that is an op­ti­miser in other con­text then it of­course might not be an op­ti­miser for this game.”

The prob­lem with this logic is the as­sump­tion that there is a “re­sult” of 0. While it’s cer­tainly true that an “idler” will ob­tain an ac­tual value at some point, so we can as­sess how they have done, there will never be a point in time that we can as­sess the on­goer. If we change the crite­ria and say that we are go­ing to as­sess at a point in time then the on­goer can sim­ply stop then and ob­tain the high­est pos­si­ble util­ity. But time never ends, and we never mark the on­goer’s home­work, so to say he has a util­ity of 0 at the end is non­sense, be­cause there is, by defi­ni­tion, no end to this sce­nario.

Essen­tially, if you in­clude in­finity in a max­imi­sa­tion sce­nario, ex­pect odd re­sults.

• “Keep go­ing un­til all that’s left is one ex­tra ap­ple, and the ra­tio­nal thing to do is to wait for­ever for an ap­ple you’ll never end up with”—that doesn’t re­ally fol­low. You have to get the Ap­ple and exit the time loop at some point or you never get any­thing.

And the in­finite time you have to spend to get that ap­ple, mul­ti­plied by the zero cost of the time, is...?

the sce­nario speci­fi­cally re­quires you to ter­mi­nate in or­der to gain any util­ity.

Your mor­tal­ity bias is show­ing. “You have to wait an in­finite amount of time” is only a mean­ingful ob­jec­tion when that costs you some­thing.

• How would you rate your maths abil­ity?

• Bet­ter than your philo­sophic abil­ity.

I can give you solu­tions for all your sam­ple prob­lems. The ap­ple-swap­ping prob­lem is a pris­oner’s dilemma; agree to split the utilon and get out. The biggest-num­ber prob­lem can be eas­ily re­solved by step­ping out­side the prob­lem frame­work with a sim­ple pyra­mid scheme (cre­ate enough utilons to cre­ate X more en­tities who can cre­ate util­ity; each en­tity then cre­ates enough util­ity to make X en­tities plus pay its cre­ator three times its cre­ation cost. Creator then spends two thirds of those utilons cre­at­ing new en­tities, and the re­main­ing third on it­self. Every en­tity en­gages in this scheme, en­sur­ing ex­po­nen­tially-in­creas­ing util­ity for ev­ery­body. Ad­just costs and pay­outs how­ever you want, in­finite util­ity is in­finite util­ity.) There are side­ways solu­tions for just about any prob­lem.

The prob­lem isn’t that any of your lit­tle sam­ple prob­lems don’t have solu­tions, the prob­lem is that you’ve already care­fully elimi­nated all the solu­tions you can think of, and will keep elimi­nat­ing solu­tions un­til no­body can think of a solu­tion—if I sug­gested the pyra­mid scheme, I’m sure you’d say I’m not al­lowed to cre­ate new en­tities us­ing my utilons, be­cause I’m break­ing what your thought ex­per­i­ment was in­tended to con­vey and just show­ing off.

I by­passed all of that and got to the point—you’re not crit­i­ciz­ing ra­tio­nal­ity for its failure to func­tion in this uni­verse, you’re crit­i­ciz­ing ra­tio­nal­ity for its be­hav­ior in rad­i­cally differ­ence uni­verses and the failure of that be­hav­ior to con­form to ba­sic san­ity-checks that only make sense in the uni­verse you your­self hap­pen to oc­cupy.

Ra­tion­al­ity be­longs to the uni­verse. In a bizarre and in­sane uni­verse, ra­tio­nal be­hav­ior is bizarre and in­sane, as it should be.

• Sorry, I was be­ing rude then.

The prob­lem is: 1) 0 times in­finity is un­defined not 0 2) You are talk­ing about in­finity as some­thing that can be reached, when it is only some­thing that can be ap­proached.

Th­ese are both very well known math­e­mat­i­cal prop­er­ties.

“If I sug­gested the pyra­mid scheme, I’m sure you’d say I’m not al­lowed to cre­ate new en­tities us­ing my utilons”—If you read Richard Ken­nawy’s com­ment—you’ll see that util­ions are not what you think that they are.

“The ap­ple-swap­ping prob­lem is a pris­oner’s dilemma; agree to split the utilon and get out.”—You may want to read this link. “Like­wise, peo­ple who re­sponds to the Trol­ley prob­lem by say­ing that they would call the po­lice are not talk­ing about the moral in­tu­itions that the Trol­ley prob­lem in­tends to ex­plore. There’s noth­ing wrong with you if those prob­lems are not in­ter­est­ing to you. But fight­ing the hy­po­thet­i­cal by challeng­ing the premises of the sce­nario is ex­actly the same as say­ing, “I don’t find this topic in­ter­est­ing for what­ever rea­son, and wish to talk about some­thing I am in­ter­ested in.”″

• 1) 0 times in­finity is un­defined not 0

Cor­rect. Now, ob­serve that’s you’ve cre­ated mul­ti­ple prob­lems with mas­sive “Un­defined” where any op­ti­miza­tion is sup­posed to take place, and then claimed you’ve proven that op­ti­miza­tion is im­pos­si­ble.

You are talk­ing about in­finity as some­thing that can be reached, when it is only some­thing that can be ap­proached.

No, I am not. I never as­sume any­body ends up with the ap­ple/​utilon, for ex­am­ple. There’s just never a point where it makes sense to stop, so you should never stop. If this doesn’t make sense to you and offends your sen­si­bil­ities, well, quit con­struct­ing non­sen­si­cal sce­nar­ios that don’t match the re­al­ity you un­der­stand.

If you read Richard Ken­nawy’s com­ment—you’ll see that util­ions are not what you think that they are.

They’re not any­thing at all, which was my point about you let­ting ab­stract things do all your heavy lift­ing for you.

“The ap­ple-swap­ping prob­lem is a pris­oner’s dilemma; agree to split the utilon and get out.”—You may want to read this link. “Like­wise, peo­ple who re­sponds to the Trol­ley prob­lem by say­ing that they would call the po­lice are not talk­ing about the moral in­tu­itions that the Trol­ley prob­lem in­tends to ex­plore. There’s noth­ing wrong with you if those prob­lems are not in­ter­est­ing to you. But fight­ing the hy­po­thet­i­cal by challeng­ing the premises of the sce­nario is ex­actly the same as say­ing, “I don’t find this topic in­ter­est­ing for what­ever rea­son, and wish to talk about some­thing I am in­ter­ested in.”″

• Very closely re­lated: Stu­art Arm­strong’s Nat­u­ral­ism ver­sus un­bounded (or un­max­imis­able) util­ity op­tions from about three years ago.

I think all this amounts to is: there can be situ­a­tions in which there is no op­ti­mal ac­tion, and there­fore if we in­sist on defin­ing “ra­tio­nal” to mean “always tak­ing the op­ti­mal ac­tion” then no agent can be perfectly “ra­tio­nal” in that sense. But I don’t know of any rea­son to adopt that defi­ni­tion. We can still say, e.g., that one course of ac­tion is more ra­tio­nal than an­other, even in situ­a­tions where no course of ac­tion is most ra­tio­nal.

• “We can still say, e.g., that one course of ac­tion is more ra­tio­nal than an­other, even in situ­a­tions where no course of ac­tion is most ra­tio­nal.”—True.

“But I don’t know of any rea­son to adopt that defi­ni­tion”—perfect ra­tio­nal­ity means to me more ra­tio­nal than any other agent. I think that is a rea­son­able defi­ni­tion.

• See­ing as this is an en­tire ar­ti­cle about nit­pick­ing and math­e­mat­i­cal con­structs...

perfect ra­tio­nal­ity means to me more ra­tio­nal than any other agent. I think that is a rea­son­able defi­ni­tion.

Surely that should be “at least as ra­tio­nal as any other agent”?

• Thanks for this com­ment. I agree, but can’t be both­ered edit­ing.

• From my per­spec­tive, there’s no con­tra­dic­tion here—or at least, the con­tra­dic­tion is con­tained within a hid­den as­sump­tion, much in the same way that the “un­stop­pable force ver­sus im­mov­able ob­ject” para­dox as­sumes the con­tra­dic­tion. An “un­stop­pable force” can­not log­i­cally ex­ist in the same uni­verse as an “im­mov­able ob­ject”, be­cause the ex­is­tence of one con­tra­dicts the ex­is­tence of the other by defi­ni­tion. Like­wise, you can­not have a “util­ity max­i­mizer” in a uni­verse where there is no “max­i­mum util­ity”—and since you ba­si­cally equate “be­ing ra­tio­nal” with “max­i­miz­ing util­ity” in your post, your ar­gu­ment begs the ques­tion.

• Ok, lets say you are right that there does not ex­ist perfect the­o­ret­i­cal ra­tio­nal­ity in your hy­po­thet­i­cal game con­text with all the as­sump­tions that helps to keep the whole game stand­ing. Nice. So what?

• Then we can ask whether there are any other situ­a­tions where perfect the­o­ret­i­cal ra­tio­nal­ity is not pos­si­ble. Be­cause we are already aware that it de­pends on the rules of the game (in­stead of as­sum­ing au­to­mat­i­cally that it is always pos­si­ble).

Ex­plor­ing the bound­ary be­tween the games where perfect the­o­ret­i­cal ra­tio­nal­ity is pos­si­ble, and the games where perfect the­o­ret­i­cal ra­tio­nal­ity is im­pos­si­ble, could lead to some in­ter­est­ing the­o­ret­i­cal re­sults. Maybe.

• It is use­ful to be able to dis­miss any pre­con­cep­tions that perfect de­ci­sion­mak­ers can ex­ist, or even be rea­soned about. I think this is a very el­e­gant way of do­ing that.

• No. It just says that perfect de­ci­sion­mak­ers can’t ex­ist in a world that vi­o­lates ba­sic physics by al­low­ing peo­ple to state even big­ger num­bers with­out spend­ing ad­di­tional time. It doesn’t say that perfect de­ci­sion­mak­ers can’t ex­ist in a world that op­er­ates un­der the physics un­der which our world op­er­ates.

The fact that you can con­structe pos­si­ble world in which there are no perfect de­ci­sion­mak­ers isn’t very in­ter­est­ing.

• “World that vi­o­lates ba­sic physics”—well the laws of physics are differ­ent in this sce­nario, but I keep the laws of logic the same, which is some­thing.

“The fact that you can con­structe pos­si­ble world in which there are no perfect de­ci­sion­mak­ers isn’t very in­ter­est­ing.”

Maybe. This is just part 1 =P.

• Spoilers, haha.

I was ac­tu­ally read­ing this post and I was try­ing to find a solu­tion to the coal­i­tion prob­lem where Eliezer won­ders how ra­tio­nal agents can solve a prob­lem with the po­ten­tial for an in­finite loop, which lead me to what I’ll call the Wait­ing Game, where you can wait n units of time and gain n util­ity for any finite n, which then led me to this post.

• Sup­pose in­stead that the game is “gain n util­ity”. No need to speak the num­ber, wait n turns, or even to wait for a meat brain to make a de­ci­sion or com­pre­hend the num­ber.

I posit that a perfectly ra­tio­nal, dis­em­bod­ied agent would de­cide to se­lect an n such that there ex­ists no n higher. If there is a pos­si­ble out­come that such an agent prefers over all other pos­si­ble out­comes, then by the defi­ni­tion of util­ity such an n ex­ists.

• Not quite. There is no rea­son in­her­ent in the defi­ni­tion that util­ity has to be bounded.

• I’m not con­vinced. It takes mas­sive amounts of ev­i­dence to con­vince me that the offers in each of your games are sincere and ac­cu­rate. In par­tic­u­lar it takes an in­finite amount of ev­i­dence to prove that your agents can keep hand­ing out in­creas­ing util­ity/​tripling/​what­ever. When some­thing in­cred­ible seems to hap­pen, fol­low the prob­a­bil­ity.

I’m re­minded of the two-en­velope game, where seem­ingly the player can get more and more money(/​util­ity) by swap­ping en­velopes back and forth. Of course the solu­tion is clear if you as­sume (any!) prior on the money in the en­velopes, and the same is hap­pen­ing if we start think­ing about the pow­ers of your game hosts.

• “It takes mas­sive amounts of ev­i­dence to con­vince me that the offers in each of your games are sincere and ac­cu­rate.”—Again, this only works if you as­sume we are mod­el­ling the real world, not perfect ce­les­tial be­ings with perfect knowl­edge. I have made no claims about whether perfect the­o­ret­i­cal ra­tio­nal­ity can ex­ist in the­ory in a world with cer­tain “re­al­ism” con­straints, just that if logic is the only con­straint, perfect ra­tio­nal­ity doesn’t ex­ist in gen­eral.

• I must ad­mit that I am now con­fused about the goal of your post. The words ‘perfect ce­les­tial be­ings with perfect knowl­edge’ sound like they mean some­thing, but I’m not sure if we are try­ing to at­tach the same mean­ing to these words. To most peo­ple ‘un­limited’ means some­thing like ‘more than a few thou­sand’, i.e. re­ally large, but for your para­doxes you need ac­tual math­e­mat­i­cal un­bound­ed­ness (or for the ex­am­ple with the 100, ar­bi­trary ac­cu­racy). I’d say that if the clos­est coun­terex­am­ple to the ex­is­tence of ‘ra­tio­nal­ity’ is a world where be­ings are no longer limited by phys­i­cal con­straints (oth­er­wise this would provide rea­son­able up­per bounds on this util­ity?) on ei­ther side of the scale (in­finitely high util­ity along with in­finitely high ac­cu­racy, so no atoms?), where for some rea­son one of such be­ings goes around dis­tribut­ing free utils and the other has in­finitely much ev­i­dence that this offer is sincere, we’re pretty safe. Or am I mi­s­un­der­stand­ing some­thing?

I think the bot­tom line is that ‘un­bounded’, in­stead of ‘re­ally frickin large’, is a tough bar to pass and it should not care­lessly be as­sumed in hy­po­thet­i­cals.

• Well, the idea be­hind “perfect ce­les­tial be­ings” kind of is to ig­nore phys­i­cal con­straints.

“I think the bot­tom line is that ‘un­bounded’, in­stead of ‘re­ally frickin large’, is a tough bar to pass and it should not care­lessly be as­sumed in hy­po­thet­i­cals”—Why? I haven’t ac­tu­ally claimed the the non-ex­is­tence of perfect ra­tio­nal­ity within the hy­po­thet­i­cal leads to any real world con­se­quences as of yet. Ar­gu­ing against an ar­gu­ment I haven’t made does noth­ing.

• 5 Jan 2016 1:03 UTC
3 points

This seems like an­other in a long line of prob­lems that come from as­sum­ing un­bounded util­ity func­tions.

Edit:The sec­ond game sounds a lot like the St. Peters­burg para­dox.

• Thanks for bring­ing this up. That isn’t quite the is­sue here though. Imag­ine that you can name any num­ber less than 100 and you gain that much util­ity, but you can’t name 100 it­self. Fur­ther­more, there is a de­vice that com­pen­sates you for any time spent speak­ing with some­thing worth equiv­a­lent util­ity. So whether you name 99 or 99.9 or 99.99… there’s always an­other agent more ra­tio­nal than you.

• Once you make that change, you’re get­ting into coastline para­dox ter­ri­tory. I don’t think that nec­es­sar­ily is a para­dox re­lated speci­fi­cally to de­ci­sion the­ory—it’s more of a prob­lem with our math sys­tem and the trou­ble with rep­re­sent­ing in­fin­tes­i­mals.

• It’s not a prob­lem with the math sys­tem. It is part of the situ­a­tion that you aren’t al­lowed to say 100 minus delta where delta is in­finites­i­mally small. In fact, we can re­strict it fur­ther and rule that the gamemaker will only ac­cept the num­ber if you list out the digits (and the dec­i­mal point if there is one). What’s wrong with perfect ra­tio­nal­ity not ex­ist­ing? On the other side of the ques­tion, on what ba­sis do we be­lieve that perfect ra­tio­nal­ity does ex­ist?

• 5 Jan 2016 3:31 UTC
2 points
Parent

I ac­tu­ally don’t be­lieve that perfect ra­tio­nal­ity does ex­ist—but in this case, I think the whole con­cept of “perfect” is flawed for this prob­lem. You can use the same ar­gu­ment to prove that there’s no perfect car­tog­ra­pher, no perfect shot­put­ter, no perfect (in­sert any­thing where you’re try­ing to get as close as you can to a num­ber with­out touch­ing it).

As I said, I don’t think it’s prov­ing any­thing spe­cial about ra­tio­nal­ity—it’s just that this a prob­lem taht we don’t have good lan­guage to dis­cuss.

• “You can use the same ar­gu­ment to prove that there’s no perfect car­tog­ra­pher, no perfect shot­put­ter, no perfect (in­sert any­thing where you’re try­ing to get as close as you can to a num­ber with­out touch­ing it).”—Why is that a prob­lem? I don’t think that I am prov­ing too much. Do you have an ar­gu­ment that a perfect shot­put­ter or perfect car­tog­ra­pher does ex­ist?

“As I said, I don’t think it’s prov­ing any­thing spe­cial about ra­tio­nal­ity”—I claim that if you sur­veyed the mem­bers of Less Wrong, at least 20% would claim that perfect the­o­ret­i­cal ra­tio­nal­ity ex­ists (my guess for ac­tual per­centage would be 50%). I main­tain that in light of these re­sults, this po­si­tion isn’t vi­able.

“We don’t have good lan­guage to dis­cuss.”—Could you clar­ify what the prob­lem with lan­guage is?

• 5 Jan 2016 3:52 UTC
0 points
Parent

What is perfect ra­tio­nal­ity in the con­text of an un­bounded util­ity func­tion?

• Con­sider the case where util­ity ap­proaches 100. The util­ity func­tion isn’t bounded, so the is­sue is some­thing else.

• 5 Jan 2016 4:09 UTC
0 points
Parent

It’s still some weird defi­ni­tions of perfec­tion when you’re deal­ing with in­fini­ties or in­finites­i­mals.

• Maybe it is weird, but noth­ing that can fairly be called perfec­tion ex­ists in this sce­nario, even if this isn’t a fair de­mand.

• There ex­ists an ir­ra­tional num­ber which is 100 minus delta where delta is in­finites­i­mally small. In my ce­les­tial lan­guage we call it “Bob”. I choose Bob. Also I name the per­son who rec­og­nizes that the in­crease in util­ity be­tween a 9 in the google­plex dec­i­mal place and a 9 in the google­plex+1 dec­i­mal place is not worth the time it takes to con­sider its value, and who there­fore goes out to spend his util­ity on black­jack and hook­ers dis­plays greater ra­tio­nal­ity than the per­son who does not.

Se­ri­ously, though, isn’t this more of an in­finity para­dox rather than an in­dict­ment on perfect ra­tio­nal­ity? There are ar­eas where the abil­ity to math­e­mat­i­cally calcu­late breaks down, ie naked sin­gu­lar­i­ties, Uncer­tainty Prin­ci­ple, as well as in­finity. Isn’t this more the is­sue at hand: that we can’t be perfectly ra­tio­nal where we can’t calcu­late pre­cisely?

• I didn’t spec­ify in the origi­nal prob­lem how the num­ber has to be speci­fied, which was a mis­take. There is no rea­son why the gamemaker can’t choose to only award util­ity for num­bers pro­vided in dec­i­mal no­ta­tion, just as any other com­pe­ti­tion has rules.

“Also I name the per­son who rec­og­nizes that the in­crease in util­ity be­tween a 9 in the google­plex dec­i­mal place and a 9 in the google­plex+1 dec­i­mal place is not worth the time it takes to con­sider its value”—we are as­sum­ing ei­ther a) an ab­stract situ­a­tion where there is zero cost of any kind of nam­ing ex­tra digits or b) the gamemaker com­pen­sates the in­di­vi­d­ual for the ex­tra time and effort re­quired to say longer num­bers.

If there is a prob­lem here, it cer­tainly isn’t that we can’t calcu­late pre­cisely. For each num­ber, we know ex­actly how much util­ity it gives us.

EDIT: Fur­ther 10-delta is not nor­mally con­sid­ered a num­ber. I imag­ine that some peo­ple might in­clude x as a num­ber, but they aren’t defin­ing the game, so num­ber means what math­e­mat­i­ci­ans in our so­ciety typ­i­cally mean by (real) num­ber.

• I’m just not con­vinced that you’re say­ing any­thing more than “Num­bers are in­finite” and find­ing a log­i­cal para­dox within. You can’t state the high­est num­ber be­cause it doesn’t ex­ist. If you pos­tu­late a high­est util­ity which is equal in value to the high­est num­ber times util­ity 1 then you have pos­tu­lated a util­ity which doesn’t ex­ist. I can not chose that which doesn’t ex­ist. That’s not a failure of ra­tio­nal­ity on my part any more than Achilles in­abil­ity to catch the tur­tle is a failure of his abil­ity to di­vide dis­tances.

I see I made Bob un­nec­es­sar­ily com­pli­cated. Bob = 99.9 Re­peat­ing (sorry don’t know how to get a vin­cu­lum over the .9) This is a num­ber. It ex­ists.

• I see I made Bob un­nec­es­sar­ily com­pli­cated. Bob = 99.9 Re­peat­ing (sorry don’t know how to get a vin­cu­lum over the .9) This is a num­ber. It ex­ists.

It is a num­ber, it is also known as 100, which we are ex­plic­itly not al­lowed to pick (0.99 re­peat­ing = 1 so 99.99 re­peat­ing = 100).

In any case, I think case­bash suc­cess­fully speci­fied a prob­lem that doesn’t have any op­ti­mal solu­tions (which is definitely in­ter­est­ing) but I don’t think that is a prob­lem for perfect ra­tio­nal­ity any­more than prob­lems that have more than one op­ti­mal solu­tion are a prob­lem for perfect ra­tio­nal­ity.

• I was born a non-Archimedean and I’ll die a non-Archimedean.

“0.99 re­peat­ing = 1” I only ac­cept that kind of talk from peo­ple with the gump­tion to ad­mit that the quo­tient of any num­ber di­vided by zero is in­finity. And I’ve got col­lege calcu­lus and 25 years of not do­ing much math­e­mat­i­cal think­ing since then to back me up.

I’ll show my­self out.

• I’m kind of defin­ing perfect ra­tio­nal­ity as the abil­ity to max­imise util­ity (more or less). If there are mul­ti­ple op­ti­mal solu­tions, then pick­ing any one max­imises util­ity. If there is no op­ti­mal solu­tion, then pick­ing none max­imises util­ity. So this is prob­le­matic for perfect ra­tio­nal­ity as defined as util­ity max­imi­sa­tion, but if you dis­agree with the defi­ni­tion, we can just taboo “perfect ra­tio­nal­ity” and talk about util­ity max­imi­sa­tion in­stead. In ei­ther case, this is some­thing peo­ple of­ten as­sume ex­ists with­out even re­al­is­ing that they are mak­ing an as­sump­tion.

• That’s fair, I tried to for­mu­late a bet­ter defi­ni­tion but couldn’t im­me­di­ately come up with any­thing that sidesteps the is­sue (with­out ex­plic­itly men­tion­ing this class of prob­lems).

When I taboo perfect ra­tio­nal­ity and in­stead just ask what the cor­rect course of ac­tion is, I have to agree that I don’t have an an­swer. In­tu­itive an­swers to ques­tions like “What would I do if I ac­tu­ally found my­self in this situ­a­tion?” and “What would the av­er­age in­tel­li­gent per­son do?” are un­satis­fy­ing be­cause they seem to rely on im­plicit costs to com­pu­ta­tional power/​time.

On the other hand I can also not gen­er­al­ize this prob­lem to more prac­ti­cal situ­a­tions (or find a similar prob­lem with­out op­ti­mal solu­tion that would be ap­pli­ca­ble to re­al­ity) so there might not be any prac­ti­cal differ­ence be­tween a perfectly ra­tio­nal agent and an agent that takes the op­ti­mal solu­tion if there is one and ex­plodes vi­o­lently if there isn’t one. Maybe the solu­tion is to sim­ply ex­clude prob­lems like this when talk­ing about ra­tio­nal­ity, un­satis­fy­ing as it may be.

In any case, it is an in­ter­est­ing prob­lem.

• If there is no op­ti­mal solu­tion, then pick­ing none max­imises util­ity.

This state­ment is not nec­es­sar­ily true when there is no op­ti­mal solu­tion be­cause the solu­tions are part of an in­finite set of solu­tions. That is, it is not true in ex­actly the situ­a­tion de­scribed in your prob­lem.

• Sorry, that was badly phrased. It should have been: “If there is no op­ti­mal solu­tion, then no mat­ter what solu­tion you pick you won’t be able to max­imise util­ity”

• Re­gard­less of what num­ber you choose, there will be an­other agent who chooses a higher num­ber than you and hence who does bet­ter at the task of util­ity op­ti­mis­ing than you do. If “perfectly ra­tio­nal” means perfect at op­ti­mis­ing util­ity (which is how it is very com­monly used), then such a perfect agent does not ex­ist. I can see the ar­gu­ment for low­ing the stan­dards of “perfect” to some­thing achiev­able, but low­er­ing it to a finite num­ber would re­sult in agents be­ing able to out­perform a “perfect” agent, which would be equally con­fus­ing.

Per­haps the solu­tion is to taboo the word “ra­tio­nal”. It seems like you agree that there does not ex­ist an agent that scores max­i­mally. Peo­ple of­ten talk about util­ity-max­imis­ing agents, which as­sumes it is pos­si­ble to have an agent which max­imises util­ity, which isn’t true for some situ­a­tions. That the as­sump­tion I am try­ing to challenge re­gard­less of whether we la­bel it perfect ra­tio­nal­ity or some­thing else.

• Let’s taboo “perfect”, and “util­ity” as well. As I see it, you are look­ing for an agent who is ca­pa­ble of choos­ing The High­est Num­ber. This num­ber does not ex­ist. There­fore it can not be cho­sen. There­fore this agent can not ex­ist. Be­cause num­bers are in­finite. In­finity para­dox is all I see.

Alter­nately, let­ting “util­ity” back in, in a uni­verse of finite time, mat­ter, and en­ergy, there does ex­ist a max­i­mum finite util­ity which is the sum to­tal of the time, mat­ter, and en­ergy in the uni­verse. There will be an num­ber which cor­re­sponds to this. Your op­po­nent can choose a num­ber higher than this but he will find the util­ity he seeks does not ex­ist.

• Alter­nately, let­ting “util­ity” back in, in a uni­verse of finite time, mat­ter, and en­ergy, there does ex­ist a max­i­mum finite util­ity which is the sum to­tal of the time, mat­ter, and en­ergy in the uni­verse.

Why can’t my util­ity func­tion be:

• 0 if I don’t get ice cream

• 1 if I get vanilla ice cream

• in­finity if I get choco­late ice cream

?

I.e. why should we for­bid a util­ity func­tion that re­turns in­finity for cer­tain sce­nar­ios, ex­cept in­so­far that it may lead to the types of prob­lems that the OP is wor­ry­ing about?

• I was bring­ing the ex­am­ple into the pre­sumed finite uni­verse in which we live, where Max­i­mum Utility = The En­tire Uni­verse. If we are dis­cussing a finite-quan­tity prob­lem than in­finite quan­tity is ipso facto ruled out.

• I think Nebu was mak­ing the point that while we nor­mally use util­ity to talk about a kind of ab­stract gain, com­put­ers can be pro­grammed with an ar­bi­trary util­ity func­tion. We would gen­er­ally put cer­tain re­straints on it so that the com­puter/​robot would be­have con­sis­tently, but those are the only limi­ta­tion. So even if there does not ex­ist such a thing as in­finite util­ity, a ra­tio­nal agent may still be re­quired to solve for these sce­nar­ios.

• I guess I’m ask­ing “Why would a finite-uni­verse nec­es­sar­ily dic­tate a finite util­ity score?”

In other words, why can’t my util­ity func­tion be:

• 0 if you give me the en­tire uni­verse minus all the ice cream.

• 1 if you give me the en­tire uni­verse minus all the choco­late ice cream.

• in­finity if I get choco­late ice cream, re­gard­less of how much choco­late ice cream I re­ceive, and re­gard­less of whether the rest of the uni­verse is in­cluded with it.

• “You are look­ing for an agent who is ca­pa­ble of choos­ing The High­est Num­ber”—the agent wants to max­imise util­ity, not to pick the high­est num­ber for its own sake, so that is mis­rep­re­sent­ing my po­si­tion. If you want to taboo util­ity, let’s use the word “lives saved” in­stead. Any­way, you say “There­fore this agent (the perfect life max­imis­ing agent) can not ex­ist”, which is ex­actly what I was con­clud­ing. Con­clud­ing the ex­act same thing as I con­cluded, sup­ports my ar­gu­ment, it doesn’t con­tra­dict it like you seem to think it does.

“Alter­nately, let­ting “util­ity” back in, in a uni­verse of finite time, mat­ter, and en­ergy, there does ex­ist a max­i­mum finite util­ity”—my ar­gu­ment is that there does not ex­ist perfect ra­tio­nal­ity within the imag­ined in­finite uni­verse. I said noth­ing about the ac­tual, ex­ist­ing uni­verse.

• Sorry, I missed that you pos­tu­lated an in­finite uni­verse in your game.

I don’t be­lieve I am mis­rep­re­sent­ing your po­si­tion. “Max­i­miz­ing util­ity” is achieved by-, and there­fore can be defined as- “choos­ing the high­est num­ber”. The wants of the agent need not be con­sid­ered. “Choos­ing the high­est num­ber” is an ex­am­ple of “do­ing some­thing im­pos­si­ble”. I think your ar­gu­ment breaks down to “An agent who can do the im­pos­si­ble can not ex­ist.” or “It is im­pos­si­ble to do the im­pos­si­ble”. I agree with this state­ment, but I don’t think it tells us any­thing use­ful. I think, but I haven’t thought it out fully, that it is the con­cept of in­finity that is trip­ping you up.

• What you’ve done is take my ar­gu­ment and trans­form it into an equiv­a­lent ob­vi­ous state­ment. That isn’t a counter-ar­gu­ment. In fact, in math­e­mat­ics, it is a method of prov­ing a the­o­rem.

If you read the other com­ments, then you’ll see that other peo­ple dis­agree with what I’ve said (and in a differ­ent man­ner than you), so I’m not just stat­ing some­thing ob­vi­ous that ev­ery­one already knows and agrees with.

• “What you’ve done is take my ar­gu­ment and trans­form it into an equiv­a­lent ob­vi­ous state­ment. That isn’t a counter-ar­gu­ment. In fact, in math­e­mat­ics, it is a method of prov­ing a the­o­rem. If you read the other com­ments, then you’ll see that other peo­ple dis­agree with what I’ve said” You’re wel­come? Feel free to make use of my proof in your con­ver­sa­tions with those guys. It looks pretty solid to me.

If a Perfect Ra­tional Agent is one who can choose Max­i­mum Finite Utility. And Utility is nu­mer­i­cally quan­tifi­able and ex­ists in in­finite quan­tities. And the Agent must choose the quan­tity of Utility by finite num­ber. Then no such agent can ex­ist. There­fore a Perfect Ra­tional Agent does not ex­ist in all pos­si­ble wor­lds.

I sup­pose I’m agree­ing but unim­pressed. Might could be this is the wrong web­site for me. Any thought ex­per­i­ment in­volv­ing in­finity does run the risk of sound­ing dan­ger­ously close to The­ol­ogy to my ears. An­gels on pin­heads and such. I’m not from around here and only dropped in to ask a spe­cific ques­tion el­se­where. Cheers.

• “Lives saved” is finite within a given light cone.

• A very spe­cific prop­erty of our uni­verse, but not uni­verses in gen­eral.

• There ex­ists an ir­ra­tional num­ber which is 100 minus delta where delta is in­finites­i­mally small.

Just as an aside, no there isn’t. In­finites­i­mal non-zero num­bers can be defined, but they’re “hy­per­re­als”, not ir­ra­tionals.

• An up­date to this post

It ap­pears that this is­sue has been dis­cussed be­fore in the thread Nat­u­ral­ism ver­sus un­bounded (or un­max­imis­able) util­ity op­tions. The dis­cus­sion there didn’t end up draw­ing the con­clu­sion that perfect ra­tio­nal­ity doesn’t ex­ist, so I be­lieve this cur­rent thread adds some­thing new.

In­stead, the ear­lier thread con­sid­ers the Heaven and Hell sce­nario where you can spend X days in Hell to get the op­por­tu­nity to spend 2X days in Heaven. Most of the dis­cus­sion on that thread was re­lated to the limit of how many days an agent count so as to exit at some point. Stu­art Arm­strong also comes up with the same solu­tion for demon­strat­ing that this prob­lem isn’t re­lated to un­bounded util­ity.

Qiaochu Yaun sum­marises one of the key take­aways: “This isn’t a para­dox about un­bounded util­ity func­tions but a para­dox about how to do de­ci­sion the­ory if you ex­pect to have to make in­finitely many de­ci­sions. Be­cause of the pos­si­ble failure of the abil­ity to ex­change limits and in­te­grals, the ex­pected util­ity of a se­quence of in­finitely many de­ci­sions can’t in gen­eral be com­puted by sum­ming up the ex­pected util­ity of each de­ci­sion sep­a­rately.”

Cu­dos to An­dreas Giger for notic­ing what most of the com­men­ta­tors seemed to miss: “How can util­ity be max­imised when there is no max­i­mum util­ity? The an­swer of course is that it can’t.” This is in­cred­ibly close to stat­ing that perfect ra­tio­nal­ity doesn’t ex­ist, but it wasn’t ex­plic­itly stated, only im­plied.

Fur­ther, Wei Dai’s com­ment on a ran­domised strat­egy that ob­tains in­finite ex­pected util­ity is an in­ter­est­ing prob­lem that will be ad­dressed in my next post.

• Okay, so if by ‘perfect ra­tio­nal­ity’ we mean “abil­ity to solve prob­lems that don’t have a solu­tion”, then I agree, perfect ra­tio­nal­ity is not pos­si­ble. Not sure if that was your point.

• I’m not ask­ing you, for ex­am­ple, to make a word out of the two let­ters Q and K, or to write a pro­gram that will de­ter­mine if an ar­bi­trary pro­gram halts.

Where ra­tio­nal­ity fails if that there is always an­other per­son who scores higher than you and there was noth­ing stop­ping you from scor­ing the same score or higher. Such a pro­gram is more ra­tio­nal than you in that situ­a­tion and there is an­other pro­gram more ra­tio­nal than them un­til in­finity. That there is no max­i­mally ra­tio­nal pro­gram, only suc­ces­sively more ra­tio­nal pro­grams is a com­pletely ac­cu­rate way of char­ac­ter­is­ing that situation

• Seems like you are ask­ing me to (or at least judg­ing me as ir­ra­tional for failing to) say a finite num­ber such that I could not have said a higher num­ber de­spite hav­ing un­limited time and re­sources. That is an im­pos­si­ble task.

• I’m ar­gu­ing against perfect ra­tio­nal­ity as defined as the abil­ity to choose the op­tion that max­imises the agents util­ity. I don’t be­lieve that this at all an un­usual way of us­ing this term. But re­gard­less, let’s taboo perfect ra­tio­nal­ity and talk about util­ity max­imi­sa­tion. There is no util­ity max­imiser for this sce­nario be­cause there is no max­i­mum util­ity that can be ob­tained. That’s all that I’m say­ing, noth­ing more noth­ing less. Yet, peo­ple of­ten as­sume that such a perfect max­imiser (aka perfectly ra­tio­nal agent) ex­ists with­out even re­al­is­ing that they are mak­ing an as­sump­tion.

• Cu­dos to An­dreas Giger for notic­ing what most of the com­men­ta­tors seemed to miss: “How can util­ity be max­imised when there is no max­i­mum util­ity? The an­swer of course is that it can’t.” This is in­cred­ibly close to stat­ing that perfect ra­tio­nal­ity doesn’t ex­ist, but it wasn’t ex­plic­itly stated, only im­plied.

I think the key is in­finite vs finite uni­verses. Any con­ceiv­able finite uni­verse can be ar­ranged in a finite num­ber of states, one, or per­haps sev­eral of which, could be as­signed max­i­mum util­ity. You can’t do this in uni­verses in­volv­ing in­finity. So if you want perfect ra­tio­nal­ity, you need to re­duce your in­finite uni­verse to just the stuff you care about. This is doable in some uni­verses, but not in the ones you posit.

In our uni­verse, we can shave off the in­finity, since we pre­sum­ably only care about our light cone.

• Mov­ing to Dis­cus­sion.

• I’m cu­ri­ous, do you dis­agree with the post?

I be­lieve that the point be­ing made is a) over­whelming sup­ported by logic, or at the very least a log­i­cally con­sis­tent al­ter­nate view­point b) im­por­tant to ra­tio­nal­ity (by pre­vent­ing peo­ple try­ing to solve prob­lems with no solu­tion) c) over­looked in pre­vi­ous dis­cus­sion or at least un­der­de­vel­oped.

Be­cause of this I took the so­cially risky gam­bit of mov­ing a low voted (pos­i­tive at the time) post to main.

• This seems like such an ob­vi­ous re­sult, I imag­ine that there’s ex­ten­sive dis­cus­sion of it within the game the­ory liter­a­ture some­where. If any­one has a good pa­per that would be appreciated

This ap­pears to be strongly re­lated to the St. Peters­burg Para­dox—ex­cept that the prize is in util­ity in­stead of cash, and the player gets to con­trol the coin (this sec­ond point sig­nifi­cantly changes the situ­a­tion).

To sum­marise the para­dox—imag­ine a pot con­tain­ing \$2 and a perfectly fair coin. The coin is tossed re­peat­edly. Every time it lands tails, the pot is dou­bled; when it even­tu­ally lands heads, the player wins the en­tire pot. (With a fair coin, this leads to an in­finite ex­pected pay­off—of course, giv­ing the player con­trol of the coin in­val­i­dates the ex­pected-value calcu­la­tion).

Pre-ex­ist­ing ex­ten­sive dis­cus­sion prob­a­bly refer­ences (or even talks about) the St. Peters­burg Para­dox—that might be a good start­ing point to find it.

• Define “dom­i­nant de­ci­sion” as an ac­tion that no other op­tion would re­sult in big­ger util­ity.

Then we could define an agent to be perfect if it chooses the dom­i­nant de­ci­sion out of its op­tions when­ever it ex­ists.

We could also define a dom­i­nant agent whos choice is always the dom­i­nant de­ci­sion.

a dom­i­nant agent can’t play the num­ber nam­ing game whereas a perfect agent isn’t con­strained to pick a unique one.

You might be as­sum­ing that when op­tions have util­ity val­ues that are not equal then there is a dom­i­nant de­ci­sion. For finite op­tion palettes this migth be the case.

• Define a “sucker” op­tion to be a an op­tion with a lower util­ity value than a some other pos­si­ble choice.

A dom­i­nant de­ci­sion is never a sucker op­tion but a perfect agent migth end up choos­ing a sucker op­tion. In the num­ber nam­ing game ev­ery op­tion is a sucker op­tion.

Thus “win­ning” is differ­ent from “not los­ing”.

• I would ar­gue that a perfect agent can never choose a “sucker” op­tion (edit:) and still be a perfect agent. It fol­lows straight from my defi­ni­tion. Of course, if you use a differ­ent defi­ni­tion, you’ll ob­tain a differ­ent re­sult.

• If it’s un­winnable by de­sign, can it strictly be called a game?

• It isn’t so much that fact that you don’t ob­tain the (non-ex­is­tant) max­i­mum that is im­por­tant, it’s the fact that an­other agent beats you when noth­ing was stop­ping you beat­ing the agent.

• Is util­ity zero-sum in this sce­nario? If I’m hy­per-ul­tra-happy, and my neigh­bor is ex­tra-su­per-über-mega-happy, that does not nec­es­sar­ily mean he beat me.

• Okay, let me restate it. It’s the fact that a differ­ent ver­sion of your­self that choose the same num­ber as your neigh­bour would have done bet­ter than what you did. Noth­ing to do with zero-sum.

• Why not just pos­tu­late a uni­verse where A>B>C>A and ask the de­ci­sion maker to pick the let­ter with the high­est value? What we think of as ra­tio­nal doesn’t nec­es­sar­ily work in other uni­verses.

• Pos­tu­lat­ing a uni­verse like that is to pos­tu­late differ­ent laws of logic. I don’t think most peo­ple ex­pect ra­tio­nal­ity to work in uni­verses with differ­ent laws of logic.

What I pos­tu­lated though, main­tains the same laws of logic, but pos­tu­lates differ­ent abil­ities, such as the abil­ity to in­stantly con­ceive of and com­mu­ni­cate ar­bi­trar­ily large num­bers. This is the kind of uni­verse that ra­tio­nal­ity should still be able to func­tion in, be­cause ra­tio­nal­ity only de­pends on logic (plus some kind of goal that is taken ax­io­mat­i­cally).

Fur­ther, if you don’t want to ac­cept these abil­ities, we can imag­ine a ,mag­i­cal de­vice that com­pen­sates for any time/​effort re­quired in pick­ing a larger num­ber.

• It’s all con­nected. You prob­a­bly need differ­ent laws of logic to get a mag­i­cal de­vice or to al­low for peo­ple to “in­stantly con­ceive of and com­mu­ni­cate ar­bi­trar­ily large num­bers.” See EY’s Univer­sal Fire where he wrote “If you stepped into a world where matches failed to strike, you would cease to ex­ist as or­ga­nized mat­ter. Real­ity is laced to­gether a lot more tightly than hu­mans might like to be­lieve.”

• The laws of logic don’t pro­hibit minds with in­finite states from ex­ist­ing, nor do they pro­hibit waves with in­finite fre­quen­cies ex­ist­ing, nor eyes that can de­tect in­finite vari­a­tion in fre­quency. Th­ese aren’t prop­er­ties of our world, but they don’t con­tra­dict logic. “It’s all con­nected some­how, but I can’t show how”—seems a bit like mag­i­cal think­ing.

• The prob­lem goes away if you add finite­ness in any of a bunch of differ­ent places: re­strict agents to only out­put de­ci­sions of bounded length, or to only fol­low strate­gies of bounded length, or ex­pected util­ities are con­strained to finitely many dis­tinct lev­els. (Mak­ing util­ity a bounded real num­ber doesn’t work, but only be­cause there are in­finitely many dis­tinct lev­els close to the bound).

The prob­lem also goes away if you al­low agents to out­put a countable se­quence of suc­ces­sively bet­ter de­ci­sions, and define an op­ti­mal se­quence as one such that for any pos­si­ble de­ci­sion, a de­ci­sion at least that good ap­pears some­where in the se­quence. This seems like the most promis­ing ap­proach.

• Ex­cept that isn’t the prob­lem and this post isn’t in­tended to ad­dress prac­ti­cal­ities, so the crit­i­cism that this is un­re­al­is­tic is ir­rele­vant.

• I would like to ex­tract the mean­ing of your thought ex­per­i­ment, but it’s difficult be­cause the con­cepts therein are prob­le­matic, or at least I don’t think they have quite the effect you imag­ine.

We will define the num­ber choos­ing game as fol­lows. You name any sin­gle finite num­ber x. You then gain x util­ity and the game then ends. You can only name a finite num­ber, nam­ing in­finity is not al­lowed.

If I were asked (by whom?) to play this game, in the first place I would only be able to at­tach some prob­a­bil­ity less than 1 to the idea that the mas­ter of the game is ac­tu­ally ca­pa­ble of grant­ing me ar­bi­trar­ily as­tro­nom­i­cal util­ity, and likely to do so. A tenet of the “ra­tio­nal­ity” that you are call­ing into ques­tion is that 0 and 1 are not prob­a­bil­ities, so if you pos­tu­late ab­solute cer­tainty in your least con­ve­nient pos­si­ble world, your thought ex­per­i­ment be­comes very ob­scure.

E.g. what about a thought ex­per­i­ment in a world where 2+2=5, and also 2+2=4 as well; I might en­ter­tain such a thought ex­per­i­ment, but (ab­sent some brilli­ant in­sight which would need to be sup­plied in ad­di­tion) I would not at­tach im­por­tance to it, in com­par­i­son to thought ex­per­i­ments that take place in a world more com­pre­hen­si­ble and similar to our own.

Now when I go ahead and at­tach a prob­a­bil­ity less than 1—even if it be an ex­tremely high prob­a­bil­ity—to the idea that the game works just as de­scribed, I would be­come se­ri­ously con­fused by this game be­cause the defi­ni­tion of a util­ity func­tion is:

A util­ity func­tion as­signs nu­mer­i­cal val­ues (“util­ities”) to out­comes, in such a way that out­comes with higher util­ities are always preferred to out­comes with lower util­ities.

yet my util­ity func­tion would, ac­cord­ing to my own (meta-...) re­flec­tion, with a sep­a­rate high prob­a­bil­ity, differ from the util­ity func­tion that the game mas­ter claims I have.

To re­solve the con­fu­sion in ques­tion, I would have to (or would in other terms) re­solve con­fu­sions that have been de­scribed clearly on LessWrong and are con­sid­ered to be the point at which the firm ground of 21st cen­tury hu­man ra­tio­nal­ity meets spec­u­la­tion. So yes, our con­cept of ra­tio­nal­ity has ad­mit­ted limits; I don’t be­lieve your thought ex­per­i­ment adds a new prob­le­matic that isn’t im­plied in the Se­quences.

How ex­actly this re­sult ap­plies to our uni­verse isn’t ex­actly clear, but that’s the challenge I’ll set for the com­ments.

Bear­ing in mind that my crit­i­cism of your thought ex­per­i­ment as de­scribed stands, I’ll add that a short story I once read comes to mind. In the story, a mod­ern hu­man finds him­self in a room in which the walls are clos­ing in; in the cen­tre of the room is a model with some balls and cup-shaped hold­ers, and in the cor­ner a skele­ton of a man in knight’s ar­mour. Be­fore he is trapped and suffers the fate of his pre­de­ces­sor, he suc­cess­fully re­ar­ranges the balls into a model of the so­lar sys­tem, gain­ing util­ity be­cause he has demon­strated his in­tel­li­gence (or the sci­en­tific ad­vance­ment of his species) as the alien game mas­ter in ques­tion would have wished.

If I were pre­sented with a game of this kind, my first re­sponse would be to ne­go­ti­ate with the game mas­ter if pos­si­ble and ask him per­ti­nent ques­tions, based on the type of en­tity he ap­pears to be. If I found that it were in my in­ter­ests to name a very large num­ber, de­pend­ing on con­text I would choose from the fol­low­ing re­sponses:

• I have var­i­ous mem­o­ries of con­tem­plat­ing the vast­ness of ex­is­tence. Please read the most piquant such mem­ory, which I am sure is still en­coded in my brain, and in­ter­pret it as a num­ber. (Surely “99999...” is only one con­ve­nient way of ex­press­ing a num­ber or mag­ni­tude)

• “The num­ber of great­est mag­ni­tude that (I, you, my CEV...) (can, would...) (com­pre­hend, deem most fit­ting...)”

• May I use Google? I would like to say “three to the three...” in Knuth’s up-ar­row no­ta­tion, but am wor­ried that I will mis­spell it and thereby fail ac­cord­ing to the na­ture of your game.

• “Now when I go ahead and at­tach a prob­a­bil­ity less than 1—even if it be an ex­tremely high prob­a­bil­ity—to the idea that the game works just as de­scribed”—You are try­ing to ap­ply re­al­is­tic con­straints to a hy­po­thet­i­cal situ­a­tion that is not in­tended to be re­al­is­tic nor where there are any claims that the re­sults carry over to the real world (as of yet). Tak­ing down an ar­gu­ment I haven’t made doesn’t ac­com­plish any­thing.

The games­mas­ter has no de­sire to en­gage with any of your ques­tions or your at­tempts to avoid di­rectly nam­ing a num­ber. He sim­ply tells you to just name a num­ber.

• You are try­ing to ap­ply re­al­is­tic con­straints to a hy­po­thet­i­cal situ­a­tion that is not in­tended to be realistic

Your thought ex­per­i­ment, as you want it to be in­ter­preted, is too un­re­al­is­tic for it to im­ply a new and sur­pris­ing cri­tique of Bayesian ra­tio­nal­ity in our world. How­ever, the ti­tle of your post im­plies (at least to me) that it does form such a cri­tique.

The games­mas­ter has no de­sire to en­gage with any of your ques­tions or your at­tempts to avoid di­rectly nam­ing a num­ber. He sim­ply tells you to just name a num­ber.

If we in­ter­pret the thought ex­per­i­ment as hap­pen­ing in a world similar to our own—which I think is more in­ter­est­ing than an in­com­pre­hen­si­ble world where the 2nd law of ther­mo­dy­nam­ics does not ex­ist and the Kol­mogorov ax­ioms don’t hold by defi­ni­tion—I would be sur­prised that such a games­mas­ter would view Ara­bic nu­mer­als as the only or best way to com­mu­ni­cate an ar­bi­trar­ily large num­ber. This seems, to me, like a prim­i­tive hu­man thought that’s very limited in com­par­i­son to the con­cepts available to a su­per­in­tel­li­gence which can read a hu­man’s source code and take mea­sure­ments of the neu­rons and sub­atomic par­ti­cles in his brain. As a hu­man play­ing this game I would, un­less told oth­er­wise in no un­cer­tain terms, try to think out­side the limited-hu­man box, both be­cause I be­lieve this would al­low me to com­mu­ni­cate num­bers of greater mag­ni­tude and be­cause I would ex­pect the games­mas­ter’s mo­tive to in­clude some­thing more in­ter­est­ing, and hu­mane and sen­si­ble, than test­ing my abil­ity to re­cite digits for an ar­bi­trary length of time.

There’s a fas­ci­nat­ing ten­sion in the idea that the games­mas­ter is an FAI, be­cause he would be­stow upon me ar­bi­trary util­ity, yet he might be so un­helpful as to have me re­cite a num­ber for billions of years or more. And what if my util­ity func­tion in­cludes (time­less?) prefer­ences that in­terfere with the func­tion­ing of the games­mas­ter or the game it­self?

• “How­ever, the ti­tle of your post”—ti­tles need to be short so they can’t con­vey all the com­plex­ity of the ac­tual situ­a­tion.

“Which I think is more in­ter­est­ing”—To each their own.

• Let’s as­sume that the be­ing that is sup­posed to find a strat­egy for this sce­nario op­er­ates in a uni­verse whose laws of physics can be speci­fied math­e­mat­i­cally. Given this sce­nario, it will try to max­i­mize the num­ber it out­puts. Its out­put can­not pos­si­bly sur­pass the max­i­mum finite num­ber that can be speci­fied us­ing a string no longer than its uni­verses speci­fi­ca­tion, so it need not try to sur­pass it, but it might come pretty close. There­fore, for each such uni­verse, there is a best ra­tio­nal ac­tor.

Edit: No, wait. Umm, you might want to find the er­ror in the above rea­son­ing your­self be­fore read­ing on. Con­sider the uni­verse with an ac­tor for ev­ery nat­u­ral num­ber that always out­puts that num­ber. The above ar­gu­ment says that no ac­tor from that uni­verse could out­put a big­ger num­ber than can be speci­fied us­ing a string no longer than the laws of physics of the uni­verse, but that only goes if the laws of physics in­clude a poin­ter to that ac­tor—to ex­tract the num­ber 100 from that uni­verse, we need to know that we want to look at the hun­dredth ac­tor. But your game didn’t re­quire that: In­side the uni­verse, each ac­tor knows that it is it­self with­out any global poin­t­ers, and so there can be an in­finite hi­er­ar­chy of bet­ter-than-the-pre­vi­ous ra­tio­nal ac­tors in a finitely speci­fied uni­verse.

• Any finite uni­verse will have a best such ac­tor, but is even our uni­verse finite? Be­sides, this was pur­pose­fully set in an in­finite uni­verse.

• Finitely speci­fied uni­verse, not finite uni­verse. That said, un­til the edit I had failed to re­al­ize that the di­ag­o­nal­iza­tion ar­gu­ment I used to dis­al­low an in­finite uni­verse to con­tain an in­finite hi­er­ar­chy of finite ac­tors doesn’t work.

• For the Un­limited Swap game, are you im­plic­itly as­sum­ing that the time spent swap­ping back and forth has some small nega­tive util­ity?

• No. There’s no util­ity lost. But if you both wait for­ever, no-one gets any util­ity.

• Can you define “for­ever” in this sce­nario? I thought it was in­finite, so there is no such thing.

• You can com­mu­ni­cate any finite num­ber in­stantly. Or, in the sec­ond ver­sion, you can’t, but you are com­pen­sated for any time, but you re­ceive the util­ity when you halt.

• You are right, the­ory is over­rated. Just be­cause you don’t have a the­o­ret­i­cal jus­tifi­ca­tion for com­menc­ing an ac­tion doesn’t mean that the ac­tion isn’t the right ac­tion to take if you want to try to “win.” Of course, it is very pos­si­ble to be in a situ­a­tion where “win­ning” is in­her­ently im­pos­si­ble, in which case you could still (ra­tio­nally) at­tempt var­i­ous strate­gies that seem likely to make you bet­ter off than you would oth­er­wise be...

As a prac­tic­ing at­tor­ney, I’ve fre­quently en­coun­tered real-life prob­lems similar to the above. For ex­am­ple, in a ne­go­ti­a­tion on be­half of a client, there is of­ten what’s called a “bar­gain­ing zone” that rep­re­sents a range of op­tions for pos­si­ble “deals” that both par­ties are the­o­ret­i­cally will­ing to ac­cept. Any given “deal” would be Pareto Effi­cient, and any “deal” within the “bar­gain­ing zone,” if it takes place, would make both par­ties to the ne­go­ti­a­tion bet­ter off than they were be­fore. How­ever, it is pos­si­ble to strike a su­pe­rior deal for your client if you are more ag­gres­sive and push the terms into the “up­per” range of the bar­gain­ing zone. On the other hand, you don’t typ­i­cally know the ex­tent of the “bar­gain­ing zone” be­fore you be­gin ne­go­ti­a­tions. If you are TOO ag­gres­sive and push out­side of the range of the other party’s ac­cept­able op­tions, the other party/​coun­sel might get frus­trated with you and call off the ne­go­ti­a­tions en­tirely, in which case you will lose the deal for ev­ery­one and make your client an­gry with you.

To the ex­tent “win­ning” is pos­si­ble here, the strat­egy for at­tor­neys on both sides is to push the terms of the “deal” as close as pos­si­ble to the “edge” of what the other will ac­cept with­out push­ing too far and get­ting the talks called off. Although there are rea­son­able strate­gies to the pro­cess, very of­ten there isn’t a the­o­ret­i­cal “op­ti­mally ra­tio­nal strat­egy” for “win­ning” a ne­go­ti­a­tion—you just have to play the game and make your strate­gic de­ci­sions based on new in­for­ma­tion as it be­comes available.

• There is an op­ti­mal strat­egy for ne­go­ti­a­tion. It re­quires es­ti­mat­ing the ne­go­ti­a­tion zone of the other party and the util­ity of var­i­ous out­comes (in­clud­ing failure of ne­go­ti­a­tion).

Then it’s just a strat­egy that max­i­mizes the sum of the prob­a­bil­ity of each out­come times the util­ity thereto.

The hard parts aren’t the P(X1)U(X1) sums, it’s get­ting the P(X1) and U(X1) in the first place.

• My gut re­sponse to the un­bounded ques­tions is that a perfectly ra­tio­nal agent would already know (or have a good guess as to) the max­i­mum util­ity that it could con­ceiv­ably ex­pect to use within the limit of the ex­pected lifes­pan of the uni­verse.

There is also an eco­nomic ob­jec­tion; at some point it seems right to ex­pect the value of ev­ery utilon to de­crease in re­sponse to the ad­di­tion of more utilons into the sys­tem.

In both ob­jec­tions I’m ap­proach­ing the same thing from differ­ent an­gles: the up­per limit on the “un­bounded” util­ity in this case de­pends on how much the uni­verse can be im­proved. The ques­tion of how to achieve max­i­mum util­ity in those sce­nar­ios is malformed similarly to that of ask­ing the end state of af­fairs af­ter com­plet­ing cer­tain su­per­tasks. More con­text is needed. I sus­pect the same is also true for the Un­limited Swap sce­nario.

• The point of utilons is to scale lin­early, un­like, say dol­lars. Maybe there’s a max­i­mum util­ity that can be ob­tained, but they never scale non-lin­early. The task where you can name any num­ber be­low 100, but not 100 it­self, avoids these is­sues though.

I don’t un­der­stand your ob­jec­tion to the Un­limited Swap sce­nario, but isn’t it plau­si­ble that a perfectly ra­tio­nal agent might not ex­ist?

• The task where you can name any num­ber be­low 100, but not 100 it­self, avoids these is­sues though.

That task still has the is­sue that the agent in­curs some un­stated cost (prob­a­bly time) to keep mash­ing on the 9 key (or what­ever in­put method). At some point, the gains are nom­i­nal and the agent would be bet­ter served col­lect­ing util­ity in the way it usu­ally does. Same goes for the Un­limited Swap sce­nario: the agent could bet­ter spend its time by in­stantly tak­ing the 1 utilon and go­ing about its busi­ness as nor­mal, thus avoid­ing a stale­mate (con­di­tion where no­body gets any util­ity) with 100% cer­tainty.

Is it plau­si­ble that a perfectly ra­tio­nal agent might not ex­ist? Cer­tainly. But I hardly think these thought ex­er­cises prove that one is not pos­si­ble. Rather, they sug­gest that when work­ing with limited in­for­ma­tion we need a sane stop­ping func­tion to avoid stale­mate. Some con­di­tions have to be “good enough”… I sup­pose I ob­ject to the con­cept of “in­finite pa­tience”.

Every­thing ex­ists in context

• True, ev­ery­thing does ex­ist in con­text. And the con­text be­ing con­sid­ered here, is not the real world, but the be­havi­our in a purely the­o­ret­i­cally con­structed world. I have made no claims that it cor­re­sponds to the real world as of yet, so claiming that it doesn’t cor­re­spond to the real world is not a valid crit­i­cism.

• My crit­i­cism is that you have ei­ther set up a set of sce­nar­ios with in­suffi­cient con­text to an­swer the ques­tion of how to ob­tain max­i­mum util­ity, or de­liber­ately con­structed these sce­nar­ios such that at­tempt­ing to ob­tain max­i­mum util­ity leads to the Ac­tor spend­ing an in­finite amount of time while failing to ever com­plete the task and ac­tu­ally col­lect. You stated that un­til the speci­fi­ca­tion of the num­ber, or the back-and-forth game was com­plete no util­ity was gained. I re­sponded that the solu­tion is to not play the game, but for the ac­tor to grab as much util­ity as it could get within a cer­tain finite time limit ac­cord­ing to its stop­ping func­tion and go about its busi­ness.

I have made no claims that it cor­re­sponds to the real world as of yet...

If it does not, then what is the point? How does such an ex­er­cise help us to be “less wrong”? The point of con­struct­ing be­liefs about Ra­tional Ac­tors is to be able to pre­dict how they would be­have so we can em­u­late that be­hav­ior. By choos­ing to ex­plore a sub­ject in this con­text, you are im­plic­itly mak­ing the claim that you be­lieve it does cor­re­spond to the real world in some way. Fur­ther­more, your choice to qual­ify your state­ment with “as of yet” re­in­forces that im­pli­ca­tion. So I ask you to state your claim so we may ex­am­ine it in full con­text.

• “In­suffi­cient con­text”—the con­text is perfectly well defined. How tired do I get con­sid­er­ing large num­bers? You don’t get tired at all! What is the op­por­tu­nity cost of con­sid­er­ing large num­ber? There is no op­por­tu­nity cost at all. And so on. It’s all very well defined.

“Re­sponded that the solu­tion is to not play the game, but for the ac­tor to grab as much util­ity as it could get within a cer­tain finite time limit ac­cord­ing to its stop­ping func­tion and go about its busi­ness.”—ex­cept that’s not a sin­gle solu­tion, but mul­ti­ple solu­tions, de­pend­ing on which num­ber you stop at.

“If it does not, then what is the point?”—This is only part 1. I plan to write more on this sub­ject even­tu­ally. As an anal­ogy, a reader of a book se­ries can’t go to an au­thor and de­mand that they re­lease vol­ume 2 right now so that they can un­der­stand part 1 in its full con­text. My ob­jec­tive here is only to con­vince peo­ple of this ab­stract the­o­ret­i­cal point, be­cause I sus­pect that I’ll need it later (but I don’t know for cer­tain).

• You don’t get tired at all… there is no cost at all...

So you have de­liber­ately con­structed a sce­nario, then defined “win­ning” as some­thing for­bid­den by the sce­nario. Un­helpful.

That’s mul­ti­ple solu­tions.

You have speci­fied mul­ti­ple games. I have defined a finite set of solu­tions for each Ac­tor that can all be stated as “use the stop­ping func­tion”. If your Ac­tor has no such func­tion, it is not ra­tio­nal be­cause it can get stuck by prob­lems with the po­ten­tial to be­come un­bounded. Re­mem­ber, the Trav­el­ing Sales­man must even­tu­ally sell some­thing or all that route plan­ning is mean­ingless. This sort of thing is ex­actly what a stop­ping func­tion is for, but you seem to have writ­ten them out of the hy­po­thet­i­cal uni­verse for some (as yet un­speci­fied) rea­son.

A reader can’t go to the au­thor and de­mand vol­ume 2...

In­cor­rect. Peo­ple do it all the time, and it is now eas­ier than ever. More­over, I ob­ject to the com­par­i­son of your es­say with a book. This con­text is more like a con­ver­sa­tion than a pub­li­ca­tion. Please get to the point.

My ob­jec­tive is to con­vince peo­ple of this ab­stract the­o­ret­i­cal point...

You have done noth­ing but re­move crite­ria for stop­ping func­tions from un­bounded sce­nar­ios. I don’t be­lieve that is con­vinc­ing any­body of any­thing. I sus­pect the state­ment “not ev­ery con­ceiv­able game in ev­ery con­ceiv­able uni­verse al­lows for a stop­ping func­tion that does not per­mit some­body else to do bet­ter” would be given a non-neg­ligible prob­a­bil­ity by most of us already. That state­ment seems to be what you have been ar­gu­ing, and seems to co­in­cide with your ti­tle.

Friendly Style Note: I (just now) no­ticed that you have made some ma­jor changes to the ar­ti­cle. It might be helpful to iso­late those changes struc­turally to make them more vi­su­ally ob­vi­ous. Re­mem­ber, we may not be reread­ing the full text very of­ten, so a times­tamp might be nice too. :)

• You’ll be pleased to know that I found a style of in­di­cat­ing ed­its that I’m happy with. I reaslised that if I make the word ed­ited sub­script then it is much less ob­nox­ious, so I’ll be us­ing this tech­nique on fu­ture posts.

• That sounds like it will be much eas­ier to read. Thank you for fol­low­ing up!

• There is no need to re-read the changes to the ar­ti­cle. The changes just in­cor­po­rate things that I’ve also writ­ten in the com­ments to re­duce the chance of new com­men­ta­tors com­ing into the thread with mi­s­un­der­stand­ings I’ve clar­ified in the com­ments.

“So you have de­liber­ately con­structed a sce­nario, then defined “win­ning” as some­thing for­bid­den by the sce­nario. Un­helpful.”—As long as the sce­nario does not ex­plic­itly pun­ish ra­tio­nal­ity, it is perfectly valid to ex­pect a perfectly ra­tio­nal agent to out­perform any other agent.

“Re­mem­ber, the Trav­el­ing Sales­man must even­tu­ally sell some­thing or all that route plan­ning is mean­ingless”—I com­pletely agree with this, not stop­ping is ir­ra­tional as you gain 0 util­ity. My point was that you can’t just say, “A perfectly ra­tio­nal agent will choose an ac­tion in this set”. You have to spec­ify which ac­tion (or ac­tions) an agent could choose whilst be­ing perfectly ra­tio­nal.

“You have done noth­ing but re­move crite­ria for stop­ping func­tions from un­bounded sce­nar­ios”—And that’s a valid situ­a­tion to hand off to any so-called “perfectly ra­tio­nal agent”. If it gets beaten, then it isn’t de­serv­ing of that name.

• There is no need to re-read the changes to the ar­ti­cle...

I have been op­er­at­ing un­der my mem­ory of the origi­nal premise. I re-read the ar­ti­cle to re­fresh that mem­ory and found the changes. I would sim­ply have been hap­pier if there was an ETA sec­tion or some­thing. No big deal, re­ally.

As long as the sce­nario does not ex­plic­itly pun­ish ra­tio­nal­ity, it is perfectly valid to ex­pect a perfectly ra­tio­nal agent to out­perform any other agent.

Not so: you have gen­er­ated in­finite op­tions such that there is no se­lec­tion that can fulfill that ex­pec­ta­tion. Any agent that tries to do so can­not be perfectly ra­tio­nal since the goal as defined is im­pos­si­ble.

• Ex­actly, if you ac­cept the defi­ni­tion of a perfectly ra­tio­nal agent as a perfect util­ity max­imiser, then there’s no util­ity max­imiser as there’s always an­other agent that ob­tains more util­ity, so there is no perfectly ra­tio­nal agent. I don’t think that this is a par­tic­u­larly un­usual way of us­ing the term “perfectly ra­tio­nal agent”.

• In this con­text, I do not ac­cept that defi­ni­tion: you can­not max­i­mize an un­bounded func­tion. A Perfectly Ra­tional Agent would know that.

• And it would still get beaten by a more ra­tio­nal agent, that would be beaten by a still more ra­tio­nal agent and so on un­til in­finity. There’s a non-ter­mi­nat­ing set of in­creas­ingly ra­tio­nal agents, but no fi­nal “most ra­tio­nal” agent.

• If the PRA isn’t try­ing to “max­i­mize” an un­bounded func­tion, it can’t very well get “beaten” by an­other agent who chooses x+n be­cause they didn’t have the same goal. I re­ject, there­fore, that an agent that obeys its stop­ping func­tion in an un­bounded sce­nario may be called any more or less “ra­tio­nal” based on that rea­son only than any other agent that does the same, re­gard­less of the util­ity it may not have col­lected.

By re­mov­ing all con­straints, you have made com­par­ing re­sults mean­ingless.

• So an agent that chooses only 1 util­ity could still be a perfectly ra­tio­nal agent in your books?

• Might be. Maybe that agent’s util­ity func­tion is ac­tu­ally bounded at 1 (it’s not try­ing to max­i­mize, af­ter all). Per­haps it wants 100 util­ity, but already has firm plans to get the other 99. Maybe it chose a value at ran­dom from the range of all pos­i­tive real num­bers (dis­tributed such that the prob­a­bil­ity of choos­ing X grows pro­por­tional to X) and pre-com­mit­ted to the re­sults, thus guaran­tee­ing a stop­ping con­di­tion with un­bounded ex­pected re­turn. Since it was miss­ing out on un­bounded util­ity in any case, get­ting liter­ally any is bet­ter than none, but the differ­ence be­tween x and y is not re­ally in­ter­est­ing.

(hu­morously) Maybe it just has bet­ter things to do than mea­sur­ing its *ahem* stop­ping func­tion against the other agents.

• You’re do­ing in­finity wrong. always spec­ify it as a limit. “as X ap­proaches zero, Y grows to in­finity”. In your case, X is the cost of calcu­lat­ing a big­ger num­ber. The “more ra­tio­nal” agent sim­ply is the one that can iden­tify and com­mu­ni­cate the big­ger num­ber in time to play the game. Taken that way, it doesn’t dis­prove perfect ra­tio­nal­ity, just perfect calcu­la­tion.

Another way to look at it is “always in­clude costs”. Even the­o­ret­i­cal perfect ra­tio­nal­ity is about trade­offs, not about the re­sults of an im­pos­si­ble calcu­la­tion.

• Could you clar­ify why you think that I am do­ing in­finity wrong? I’m not ac­tu­ally us­ing in­finity, just stat­ing that you aren’t al­lowed to say in­finity, but can only choose a finite num­ber.

As stated in the ar­ti­cle, I’m con­sid­er­ing the the­o­ret­i­cal case where ei­ther a) there are no costs to iden­ti­fy­ing and com­mu­ni­cat­ing ar­bi­trar­ily large num­bers (as stated, we are con­sid­er­ing ce­les­tial be­ing not real phys­i­cal be­ings) or b) we are con­sid­er­ing real be­ings, but where any costs re­lated to the effort of iden­ti­fy­ing a larger num­ber are offset by a mag­i­cal device

I already ad­mit­ted that the real world is not like this due to as­pects such as calcu­la­tion costs. I find the idea of a pur­pose­ful the­o­ret­i­cal model be­ing wrong due to real con­straints odd. If some­one puts out a the­o­ret­i­cal situ­a­tion as mod­el­ling the real world, then that might be a valid cri­tique, but when some­one is speci­fi­cally imag­in­ing a world that be­haves differ­ently from ours there is no re­quire­ment for it to be “real”.

All I am claiming is that within at least one the­o­ret­i­cal world (which I’ve pro­vided) perfect ra­tio­nal­ity does not ex­ist. Whether or not this has any bear­ing on the real world was not dis­cussed and is left to the reader to spec­u­late on.

• You’re do­ing it wrong by try­ing to use a limit (good) with­out spec­i­fy­ing the func­tion (mak­ing it mean­ingless).

there are no costs

This is the hid­den in­finity in your ex­am­ple. There can’t be zero cost. When you eval­u­ate the marginal value of a fur­ther calcu­la­tion, you take ex­pected benefit di­vided by ex­pected cost. oops, in­finity!

Alter­nately—you hy­poth­e­size that any agent would ac­tu­ally stop calcu­lat­ing and pick a num­ber. Why not calcu­late fur­ther? If it’s costless, keep go­ing. I’m not sure in your sce­nario which in­finity wins: in­finitely small cost of calcu­la­tion or in­finite time to calcu­late. Either way, it’s not about whether perfect ra­tio­nal­ity ex­ists, it’s about which in­finity you choose to break first.

• If you keep go­ing for­ever then you never re­al­ise any gains, even if it is costless, so that isn’t the ra­tio­nal solu­tion.

“This is the hid­den in­finity in your ex­am­ple. There can’t be zero cost. When you eval­u­ate the marginal value of a fur­ther calcu­la­tion, you take ex­pected benefit di­vided by ex­pected cost. oops, in­finity!”—so let’s sup­pose I give an agent a once-off op­por­tu­nity to gain 100 util­ity for 0 cost. The agent tries to eval­u­ate if it should take this op­por­tu­nity and fails be­cause there is no cost and it ends up with an in­finity. I would ar­gue that such an agent is very far away from ra­tio­nal if it can’t han­dle this sim­ple situ­a­tion.

“You’re do­ing it wrong by try­ing to use a limit (good) with­out spec­i­fy­ing the func­tion (mak­ing it mean­ingless)”—Sorry, it still isn’t clear what you are get­ting at here. I’m not try­ing to use a limit. You are the one who is in­sist­ing that I need to use a limit to eval­u­ate this situ­a­tion. Have you con­sid­ered that there might ac­tu­ally be other ways of eval­u­at­ing the situ­a­tion? The situ­a­tion is well speci­fied. State any num­ber and re­ceive that much util­ity. If you want a util­ity func­tion, u(x)=x is it. If you’re look­ing for an­other kind of func­tion, well what kind of func­tion are you look­ing for then? Sim­ply stat­ing that I haven’t speci­fied a func­tion isn’t very clear un­less you an­swer this ques­tion.

• If it takes time, that’s a cost. In your sce­nario, an agent can keep go­ing for­ever in­stantly, what­ever that means. That’s the non­sense you need to re­solve to have a co­her­ent prob­lem. Add in a time limit and calcu­la­tion rate, and you’re back to nor­mal ra­tio­nal­ity. As the time limit or rate ap­proach in­finity, so does the util­ity.

• “Add in a time limit and calcu­la­tion rate, and you’re back to nor­mal ra­tio­nal­ity”—I am in­ten­tion­ally mod­el­ling a the­o­ret­i­cal con­struct, not re­al­ity. Claims that my situ­a­tion isn’t re­al­is­tic aren’t valid, as I have never claimed that this the­o­ret­i­cal situ­a­tion does cor­re­spond to re­al­ity. I have pur­pose­fully left this ques­tion open.

• Ai-yah. That’s fine, but please then be sure to caveat your con­clu­sion with “in this non-world...” rather than gen­er­al­iz­ing about nonex­is­tence of some­thing.

• The perfectly ra­tio­nal agent con­sid­ers all pos­si­ble differ­ent world-states, de­ter­mines the util­ity of each of them, and states “X”, where X is the util­ity of the perfect world.

For the num­ber “X+ep­silon” to have been a le­gal re­sponse, the agent would have had to been mis­taken about their util­ity func­tion or what the pos­si­ble wor­lds were.

There­fore X is the largest real num­ber.

Note that this is a con­struc­tive proof, and any at­tempt at coun­terex­am­ple should at­tempt to prove that the spe­cific X dis­cov­ered by a perfectly ra­tio­nal om­ni­scient ab­stract agent with a ge­nie. If the gen­eral solu­tion is true, it will be triv­ially true for one num­ber.

• That’s not how maths works.