# Knightian Uncertainty: Bayesian Agents and the MMEU rule

Re­cently, I found my­self in a con­ver­sa­tion with some­one ad­vo­cat­ing the use of Knigh­tian un­cer­tainty. We both agreed that there’s no point in singling out some of your un­cer­tainty as “spe­cial” un­less you treat it differ­ently.

I am un­der the im­pres­sion that most ad­vice from ad­vo­cates of Knigh­tian un­cer­tainty can be taken to heart in a Bayesian frame­work, and so I find the con­cept of “Knigh­tian un­cer­tainty” un­com­pel­ling. My friend, who I’m anonymiz­ing as “Sir Percy”, claims that he does treat Knigh­tian un­cer­tainty differ­ently from nor­mal un­cer­tainty, and so he needs to make the dis­tinc­tion. Un­like an as­piring Bayesian rea­soner, who at­tempts to max­i­mize ex­pected value, he max­i­mizes the min­i­mum ex­pected value given his Knigh­tian un­cer­tainty. This is the MMEU rule mo­ti­vated pre­vi­ously.

This sur­prised me: is it pos­si­ble for a ra­tio­nal agent to re­fuse to max­i­mize ex­pected util­ity? My re­flex­ive re­ac­tion was sim­ple:

If you’re a ra­tio­nal agent and you don’t think you’re max­i­miz­ing ex­pected util­ity, then you’ve mis­placed your “util­ity” la­bel.

That can’t be. Re­mem­ber Sir Percy’s coin toss. A coin has been tossed, and the event “H” is “the coin came up heads”. Con­sider the fol­low­ing two bets:

1. Pay 50¢ to be payed \$1.10 if H

2. Pay 50¢ to be payed \$1.10 if ¬H

I don’t know whether the coin was bi­ased, I know only that my cre­dence is in the in­ter­val `[0.4, 0.6]`. When con­sid­er­ing the first bet, I no­tice that the prob­a­bil­ity of H may be 0.4, in which case the bet is ex­pected to lose me 6¢. When con­sid­er­ing the sec­ond bet, I no­tice that the prob­a­bil­ity of H may be 0.6, in which case the bet is ex­pected to lose me 6¢. But when con­sid­er­ing both to­gether, I see that I will win 10¢ with cer­tainty. So I re­ject each bet if pre­sented in­di­vi­d­u­ally, but I will pay up to 10¢ to play the pair.

As you can see, my prefer­ences change un­der ag­glomer­a­tion of bets. It’s not pos­si­ble to view me as a Bayesian rea­soner max­i­miz­ing ex­pected util­ity, be­cause there is no cre­dence you can as­sign to H such that a Bayesian rea­soner shares my prefer­ences. I can’t be max­i­miz­ing ex­pected util­ity, no mat­ter where you put your la­bels.

My re­jec­tion was vague and un­formed at the time. I’ve since fleshed it out, and it will be pre­sented be­low. But be­fore con­tin­u­ing, see if you can spot my ar­gu­ment on your own:

My friend be­lieves that H oc­cured with prob­a­bil­ity in the in­ter­val [.4, .6]. He is un­will­ing to pay 50¢ to be payed \$1.10 if H, and he is un­will­ing to pay 50¢ to be payed \$1.10 if ¬H, but he is will­ing to pay up to 10¢ for the pair. There is no way to as­sign a cre­dence to the event H such that these ac­tions are tra­di­tion­ally con­sis­tent. Yet, if we al­low that ra­tio­nal agents can in prin­ci­ple have prefer­ences about am­bi­guity, then he is act­ing ‘ra­tio­nally’ in some sense. Is the Bayesian frame­work ca­pa­ble of cap­tur­ing agents with these prefer­ences?

Short an­swer: Yes.

A Bayesian agent can act like this, and Sir Percy did put his util­ity la­bel in the wrong place. In fact, there are two ways that a Bayesian ideal­iza­tion of Sir Percy can ex­hibit Sir Percy’s prefer­ences. In the coin toss game, at least one of two things is hap­pen­ing:

1. Sir Percy isn’t treat­ing H like an event.

2. Sir Percy isn’t treat­ing money like util­ity.

I’ll ex­plore both pos­si­bil­ities in turn, but first, let’s build a lit­tle in­tu­ition:

The prefer­ences of a ra­tio­nal agent are not sup­posed to be in­var­i­ant un­der ag­glomer­a­tion of bets. Imag­ine that there are two po­tions, a blue po­tion and a green po­tion. Each po­tion, taken alone, makes you sick. Both po­tions, taken to­gether, give you su­per­pow­ers. It may well be that you re­ject both “bet 1: 100% you drink the blue po­tion” and “bet 2: 100% you drink the green po­tion”, but that you hap­pily pay for the pair.

This is not ir­ra­tional: You could say there is no as­sign­ment of util­ity to the ac­tion “drink the blue po­tion” that makes my ac­tions con­sis­tent, but this is an er­ror. Prefer­ences are not over ac­tions, they are over out­comes. If I take bet 1, I get sick. If I take bet 2, I get sick. If I take them both to­gether, I get su­per­pow­ers.

The fact that I do not honor ag­glomer­a­tion of bets does not mean I am ir­ra­tional. Sir Percy looks ir­ra­tional not be­cause his prefer­ences di­s­obey ag­glomer­a­tion of bets, but only be­cause there’s no cre­dence which Sir Percy can as­sign to H that makes his ac­tions con­sis­tent with ex­pected money max­i­miza­tion.

This means that, if Sir Percy is tra­di­tion­ally ra­tio­nal, then he’s ei­ther not treat­ing H like an event, or he’s not treat­ing money like util­ity. And as it turns out, ideal­ized Bayesian rea­son­ers can dis­play Sir Percy’s prefer­ences. I’ll illus­trate two of them be­low.

# An­tag­o­nis­tic ambiguity

Re­mem­ber Sir Percy’s origi­nal mo­ti­va­tion for re­ject­ing both bets in­di­vi­d­u­ally in Sir Percy’s coin toss:

I’m max­i­miz­ing the min­i­mum ex­pected util­ity. Given bet (1), I no­tice that per­haps the prob­a­bil­ity of H is only 40%, in which case the ex­pected util­ity of bet (1) is −6¢, so I re­ject it. Given bet (2), I no­tice that per­haps the prob­a­bil­ity of H is 60%, in which case the ex­pected util­ity of bet (2) is −6¢, so I re­ject that too.”

No­tice how Sir Percy is act­ing here: given each bet, he as­sumes that the coin is weighted in what­ever man­ner is worst for him. Sir Percy’s `[0.4, 0.6]` cre­dence in­ter­val for H im­plies that he has seen ev­i­dence that con­vinces him the coin isn’t weighted to land H less than 40% of the time, and isn’t weighted to land H more than 60% of the time, but he can’t nar­row it down any fur­ther than that. And when he’s con­sid­er­ing bets, he acts as­sum­ing that the coin is ac­tu­ally weighted in the least con­ve­nient way.

We can de­sign a Bayesian agent that acts like this, but this Bayesian agent (who I’ll call Para­noid Perry) doesn’t treat H like an event. Rather, Perry acts as if the coin’s weight­ing isn’t cho­sen un­til af­ter Perry se­lects a bet. Then, na­ture gets to choose the weight­ing of the coin (within the con­straints set by Perry’s con­fi­dence in­ter­val), and na­ture always chooses what’s worst for Perry.

Let’s say that Perry is a perfectly ra­tio­nal Bayesian agent. If na­ture has a se­lec­tion of coins weighted from 40% heads to 60% heads, and na­ture gets to se­lect one of the coins af­ter Perry se­lects one of the bets, then re­ject­ing each of the bets in­di­vi­d­u­ally but ac­cept­ing both to­gether is pre­cisely how Perry max­i­mizes ex­pected util­ity. The game seems a bit odd un­til you re­al­ize that Perry isn’t treat­ing H as an event: Perry is act­ing as if it’s choice of bets af­fects the weight­ing of the coin.

No­tice that Perry isn’t treat­ing its `[0.4, 0.6]` cre­dence in­ter­val for H as if it’s nor­mal un­cer­tainty. Perry is treat­ing this un­cer­tainty as if it is in the world: the coin could ac­tu­ally be weighted any­where from 40% H to 60% H, and na­ture gets to choose. It’s un­clear how Perry de­cides which un­cer­tainty is in­ter­nal (sub­ject to nor­mal Bayesian rea­son­ing) and which un­cer­tainty is ex­ter­nal (re­solved ad­ver­sar­i­ally by na­ture), but this dis­tinc­tion gives rise to the differ­ence be­tween “nor­mal” un­cer­tainty and “Knigh­tian” un­cer­tainty. Given any mechanism for differ­en­ti­at­ing be­tween un­cer­tainty about what world Perry is in and vari­abil­ity that gets to be re­solved by na­ture, we can con­struct a Bayesian rea­soner that acts like Perry.

Take, for ex­am­ple, the Ells­berg urn game. Perry has some­how been con­vinced that the pro­cess of se­lect­ing balls from the urn is fair: balls are se­lected by some stochas­tic pro­cess which Na­ture can­not af­fect. How­ever, Perry also things that the urn in use gets to be cho­sen by na­ture af­ter Perry has se­lected a bet.

There are many mechanisms by which Perry could come to this state of knowl­edge. Per­haps the ball-se­lec­tion mechanism was de­signed and ver­ified by Perry, but the urn is se­lected by an un­ver­ified pro­cess that is sup­pos­edly ran­dom but which Perry has rea­son to be­lieve has ac­tu­ally been in­fected by the ne­far­i­ous spirit of the Ad­ver­sary. Given this knowl­edge, Perry knows that if it chooses bet 1b then na­ture will se­lect the urn with no black balls, and if it chooses bet 2a then na­ture will se­lect the urn with­out yel­low balls, so Perry ra­tio­nally prefers bets 1a and 2b.

Perry is averse to am­bi­guity be­cause Perry be­lieves that na­ture gets to re­solve am­bi­guity, and na­ture is work­ing against Perry. This para­noia cap­tures some of the origi­nal mo­ti­va­tion for am­bi­guity aver­sion, but it starts to seem strange un­der close ex­am­i­na­tion. For ex­am­ple, in the coin toss game, Perry acts as if na­ture gets to pick the weight­ing of the coin even af­ter the coin has been tossed. Perry be­lieves that na­ture gets to re­solve am­bi­guity, even if the am­bi­guity lies in the past.

Fur­ther­more, Perry acts as if it is cer­tain that na­ture gets to re­solve am­bi­guity dis­fa­vor­ably, and this can lead to patholog­i­cal be­hav­ior in some edge cases. For ex­am­ple, con­sider Perry rea­son­ing about the un­bal­anced ten­nis game: Perry be­lieves that one of the play­ers is far bet­ter than the other, ca­pa­ble of win­ning 99 games out of 100. But Perry doesn’t know whether An­abel or Zara is the bet­ter player. How might the con­ver­sa­tion go when a bookie ap­proaches Perry offer­ing a bet of 2:1 odds on the player of Perry’s choice?

Hello, Perry, I’m a rich ec­cen­tric bookie who likes giv­ing peo­ple money in the form of strange bets. I’d like to offer you a bet on the un­bal­anced ten­nis game. I know that you know that one of the play­ers is far bet­ter than the other, and I know you have “Knigh­tian un­cer­tainty” about which player is bet­ter. To coun­ter­mand your con­cerns, I’m go­ing to make you a very good deal: I’m will­ing to offer you a bet with 2:1 odds on the player of your choice.

“I’m sorry”, Perry re­sponds, “I can­not take that bet.”

But why not? I un­der­stand that if I came to you offer­ing a bet with 2:1 odds on An­abel, you should re­ject it and then up­date in fa­vor of An­abel be­ing the bet­ter player, un­der the as­sump­tion that I was try­ing to take ad­van­tage of your un­cer­tainty. But I’m offer­ing you a bet with 2:1 odds on the player of your choice! How can you re­ject this?

“Well, you see”, Perry re­sponds, “no mat­ter what choice I make, na­ture will re­solve my am­bi­guity against me. If I place 2:1 odds on Zara win­ning, then na­ture will de­cide that Zara is the worse player, and Zara will loose 99% of the time. But if I in­stead place 2:1 odds on Zara los­ing, then na­ture will de­cide that Zara is the bet­ter player, and Zara will win 99% of the time. Either way, I lose money.

How can that be? Surely, you be­lieve that one player is already bet­ter than the other. Your choice can’t change who is bet­ter; that was de­cided in the past! You’re act­ing like na­ture will retroac­tively make the one that you bet on be worse!

“Yes, pre­cisely”, Perry re­sponds. “See, I have Knigh­tian un­cer­tainty about which is bet­ter, and na­ture gets to re­solve Knigh­tian un­cer­tainty, re­gard­less of causal­ity. No mat­ter which player I pick, she will turn out to be the worse player.”

Real­iz­ing that this may sound crazy, Perry pauses.

“Now, I’m not sure how this hap­pens. Per­haps na­ture has acausal su­per­pow­ers, ac­tu­ally can al­ter the past. Or per­haps you simu­lated me to see who I would pick, and you’re only offer­ing me this bet be­cause you found that I would pick the wrong player. I don’t know how na­ture does it, I do know for a fact that na­ture will re­solve my am­bi­guity an­tag­o­nis­ti­cally. Of this I am cer­tain. And so I’m sorry, but I can’t take your bet.”

And if I offer you 10:1 odds on the player of your choice?

The bookie de­ject­edly de­parts.

Perry is a Bayesian agent that shares Sir Percy’s prefer­ences, thereby demon­strat­ing that the Bayesian frame­work can cap­ture agents with prefer­ences over am­bi­guity (given some method for dis­t­in­guish­ing am­bi­guity from un­cer­tainty). Any Bayesian agent that be­lieves that there is ac­tu­ally vari­abil­ity in the en­vi­ron­ment which na­ture gets to acausally ad­ver­sar­i­ally re­solve acts pre­cisely as if it is us­ing the MMEU rule.

In fact, though Perry may seem to be rea­son­ing in an odd fash­ion, Perry cap­tures much of the origi­nal mo­ti­va­tion for the MMEU rule. “I can’t get a pre­cise cre­dence”, Sir Percy would say, “I can only get a cre­dence in­ter­val, be­cause re­al­ity ac­tu­ally still has a lit­tle wig­gle room about whether the coin was weighted. So as­sum­ing the worst case, how do I max­i­mize ex­pected util­ity?”

The para­noia of Perry is a lit­tle dis­con­cert­ing, though. Is there per­haps an­other ideal­ized Bayesian agent that cap­tures Sir Perry’s prefer­ences in a less abra­sive man­ner?

# Prefer­ring the least con­ve­nient world

Allow me to in­tro­duce an­other Bayesian agent, Cau­tious Caul. Like Perry, Caul dis­t­in­guishes be­tween differ­ent types of un­cer­tainty. The method of dis­tinc­tion is left un­speci­fied, but given any means of differ­en­ti­at­ing be­tween un­cer­tainty and am­bi­guity (“Knigh­tian un­cer­tainty”), we can spec­ify an agent that acts like Caul.

Like Perry, Caul treats am­bi­guity differ­ently form nor­mal un­cer­tainty be­cause Caul be­lieves that am­bi­guity is not in­ter­nal lack of knowl­edge, but rather a fact about the ex­ter­nal world. But whereas Perry thinks that Knigh­tian un­cer­tainty de­notes vari­abil­ity in the world that is re­solved by na­ture, Caul in­stead thinks that am­bi­guity de­notes wor­ld­parts that ac­tu­ally ex­ist.

And Caul is an ex­pected util­ity max­i­mizer (and a perfect Bayesian), but Caul’s prefer­ences are defined ac­cord­ing to the least con­ve­nient world.

When faced with Sir Percy’s coin toss, Caul hon­estly be­lieves that a con­tin­u­ous set of wor­lds ex­ist, each with a coin weighted some­where be­tween 40% H and 60% H, and that in each of these wor­lds there is a sliver of Caul. But Caul’s prefer­ences are defined such that Caul only cares about the sliver in the least con­ve­nient world, the world with the worst odds.

In other words, ev­ery Caul-sliver is will­ing to trade un­limited amounts of ex­pected dol­lars in their own wor­l­part in or­der to in­crease the ex­pected dol­lars of the Caul in the least con­ve­nient wor­ld­part. Thus, Caul re­fuses the first bet in Sir Percy’s coin toss, be­cause con­di­tioned upon Caul tak­ing the first bet, Caul only cares about the wor­ld­part where the coin is weighted 40% H (and so Caul re­fuses the bet). But con­di­tioned upon Caul tak­ing the sec­ond bet, Caul only cares about the wor­ld­part where the coin is weighted 60% H (and so Caul re­fuses the bet). But when offered both bets si­mul­ta­neously, all the Caul slivers in all the wor­ld­parts will gain 10¢, so Caul takes the com­bi­na­tion.

In this case, we see why there is no Bayesian with a sin­gle cre­dence for H that can ex­hibit Caul’s prefer­ences. From Caul’s point of view, H is not a sin­gle event, it is a set of events with vary­ing cre­dence. Any Bayesian at­tempt­ing to as­sign a sin­gle cre­dence to H is ne­glect­ing a cru­cial part of (what Caul sees as) the world’s struc­ture.

Fur­ther­more, Caul also fails to treat the out­comes of the bet (in dol­lars) as util­ity: Caul only cares about the least con­ve­nient world given Caul’s ac­tion, so the util­ity that Caul gets from a bet is de­pen­dent upon Caul’s ac­tion. We can see this when imag­in­ing Caul fac­ing the Ells­berg urn game. In this sce­nario, Caul has ac­tu­ally come to be­lieve that there is a Caul-sliver fac­ing each of the sixty urns. Con­di­tioned upon tak­ing bet 1b, Caul only cares about the Caul-sliver fac­ing the urn with­out any black balls. But con­di­tioned upon tak­ing bet 2a, Caul only cares about the Caul-sliver fac­ing the urn with­out any yel­low balls. Caul’s prefer­ences de­pend upon the ac­tion which Caul chooses.

Caul is an ex­pected util­ity max­i­mizer with prefer­ences only for the wor­ld­part with the worst odds (where wor­ld­parts are de­ter­mined by am­bi­guity and odds are de­ter­mined by un­cer­tainty). Caul may ap­pear to be us­ing the MMEU rule, but call is ac­tu­ally max­i­miz­ing min­i­mum ex­pected dol­lars while max­i­miz­ing ac­tual ex­pected util­ity. This only looks like MMEU if we con­fuse the in­ner util­ity in each wor­ld­part (“dol­lars”) with the outer util­ity of Caul it­self.

To illus­trate, con­sider what hap­pens when Caul en­coun­ters our en­thu­si­as­tic bookie. As with Perry, Caul be­lieves that one of the ten­nis play­ers in the un­bal­anced game could beat the other 99 times out of 100. How­ever, Caul has Knigh­tian un­cer­tainty about whether An­abel or Zara is the su­pe­rior player. This means that Caul acu­tally be­lieves there are two wor­ld­parts, each with a Caul-sliver: one in which An­abel beats Zara 99 times out of 100, and the other in which An­abel loses to Zara 99 times out of 100.

How might Caul’s con­ver­sa­tion with the bookie go?

Ah, Caul! I have an offer for you. You know the un­bal­anced ten­nis game that is about to be played be­tween An­abel and Zara? I’d like to offer you a bet that’s heav­ily in your fa­vor: I’ll give you a bet with 2:1 odds on the player of your choice!

“Sorry”, Caul says, “can’t do that.”

Why not? Look, I’m not try­ing to screw you over here. I just re­ally like giv­ing peo­ple free money, and if I can’t give enough away, I’ll have to an­swer to my board of trustees. I se­ri­ously don’t want to mess with those folks. This re­ally is a bet in your fa­vor! It pays out 2:1. I haven’t simu­lated you, I promise!

“I’m sorry”, Caul replies, “but I can’t. See, if I place the bet on An­abel, then in the world where Zara is the bet­ter player, I ex­pect that ver­sion of Caul to lose money.”

Yes, but in the world where An­abel is the bet­ter player, that ver­sion of Caul wins more than the other ver­sion loses!

“Per­haps, but I only care about the sliver of me with the worst odds. If I place the bet on An­abel, then I care only about the Caul in the world where she’s worse. But if I place the bet on Zara, then I only care about the Caul in the world where she’s bet­ter.” Caul shrugs. “Sorry, there’s just no way I can take the bet with­out mak­ing the least con­ve­nient world worse off, and I only care about what hap­pens in the least con­ve­nient world.”

And so our hap­less bookie de­parts, fran­ti­cally search­ing for some­one to take free money, mut­ter­ing about ir­ra­tional­ity.

Caul cap­tures an­other por­tion of the mo­ti­va­tion for the MMEU rule: Caul only cares about the least con­ve­nient world. Of course, Caul only cares about the least con­ve­nient world given Caul’s nor­mal un­cer­tainty: Caul cares about the Caul-sliver with the worst odds, not with the worst out­come. To see the differ­ence, imag­ine that our bookie offered Caul a bet with 100:1 odds on An­abel. Caul would take this bet, be­cause even in the world with the worst odds the ex­pected Caul-value of this bet is 1¢ (99% of the time Zara wins and Caul loses \$1, but 1% of the time An­abel wins and Caul wins \$100).

An agent that only cared about the ac­tual worst case (rather than only the wor­ld­part with the worst odds) would re­fuse even this bet, wor­ry­ing about sce­nar­ios where An­abel is a far su­pe­rior player but loses any­way. This would be an agent in which all un­cer­tainty is “Knigh­tian”, and this would turn Caul into a max­i­mizer of min­i­mum ac­tual util­ity (in­stead of min­i­mum ex­pected util­ity), which is wildly im­prac­ti­cal (and of du­bi­ous use­ful­ness). This raises the un­com­fortable ques­tion of how Caul de­cides which un­cer­tain­ties de­note ac­tual wor­ld­parts, and which un­cer­tain­ties de­note “nor­mal” un­cer­tainty with which Caul may gam­ble in the tra­di­tional man­ner.

Perry and Caul show that it’s pos­si­ble for a Bayesian to act ac­cord­ing to Sir Percy’s prefer­ences, but they are only able to do this be­cause they be­lieve that some of their un­cer­tainty is not in­ter­nal (caused by in­com­plete in­for­ma­tion) but rather ex­ter­nal (ac­tu­ally rep­re­sented in the en­vi­ron­ment). Perry be­lieves that some of its un­cer­tainty de­notes places where Na­ture gets to choose how the world works af­ter Perry chooses an ac­tion. Caul be­lieves that some of its un­cer­tainty ac­tu­ally de­notes real wor­ld­parts. In both cases, these agents be­lieve that their am­bi­guity is part of the world, and this is the mechanism by which they can dis­play am­bi­guity aver­sion.

# Bayesian am­bi­guity aversion

Perry and Caul help us an­swer our first ques­tion.

Can an agent with prefer­ences about am­bi­guity be ‘ra­tio­nal’ in the Bayesian sense?

To this, we can now an­swer “yes”. Ideal Bayesian agents can ex­hibit the prefer­ences ad­vo­cated by Sir Percy. The next ques­tion is, is this a sane way to act? A ra­tio­nal agent can ex­hibit am­bi­guity aver­sion, and hu­mans seem to ex­hibit am­bi­guity aver­sion… so should we act like this?

Sir Percy ad­vo­cates this de­ci­sion rule. He’s put forth some in­ter­est­ing ar­gu­ments and this post has shown that such a view­point can be ra­tio­nal in a Bayesian sense. But a Bayesian can also have an anti-Lapla­cian prior, so that doesn’t speak to this view­point’s san­ity. Is the MMEU rule a sane way for hu­mans to act?

In short: No, for the same rea­son that Caul and Perry are mad. Their views are con­sis­tent, and their ac­tions are ra­tio­nal (given their pri­ors), but I am re­ally glad that I don’t have their pri­ors. But that’s a point I’ll ex­plore in more depth in the next (and fi­nal) post.

• I think the anal­y­sis in this post (and the oth­ers in the se­quence) has all been spot on, but I don’t know that it is ac­tu­ally all that use­ful. I’ll try to ex­plain why.

This is how I would steel man Sir Percy’s de­ci­sion pro­cess (stipu­lat­ing that Sir Percy him­self might not agree):

Most bets are offered be­cause the per­son offer­ing ex­pects to make a profit. And fre­quently, they are will­ing to ex­ploit in­for­ma­tion that only they have, so they can offer bets that will seem rea­son­able to me but which are ac­tu­ally un­fa­vor­able.

When I am offered a bet where there is some im­por­tant un­known fac­tor (e.g. which way the coin is weighted, or which urn I am draw­ing from), I am highly sus­pi­cious that the per­son offer­ing the bet knows some­thing that I don’t, even if I don’t know where they got their in­for­ma­tion. There­fore, I will be very re­luc­tant to take such bets

When faced with this kind of bet, a perfect bayesian would calcu­late p(bet is se­cretly un­fair | am­bigu­ous bet is offered) and use that as an in­put into their ex­pected util­ity calcu­la­tions. In al­most ev­ery situ­a­tion one might come across, that prob­a­bil­ity is go­ing to be quite high. There­fore, the gen­eral in­tu­ition of “don’t mess with am­bigu­ous bets—the other guy prob­a­bly knows some­thing you don’t” is a pretty good one.

Of course you can con­struct thought ex­per­i­ments where p(bet is se­cretly un­fair) is ac­tu­ally 0 and the in­tu­ition breaks down. But those situ­a­tions are very un­likely to come up in re­al­ity (un­less there are ac­tu­ally a lot of bizarrely gen­er­ous book­ies out there, in which case I should stop typ­ing this and go find them be­fore they run out of money). So while it is tech­ni­cally true that a perfect Bayesian would ac­tu­ally calcu­late p(bet is se­cretly un­fair | am­bigu­ous bet was offered) in ev­ery situ­a­tion with an am­bigu­ous bet, it seems like a very rea­son­able short­cut to just as­sume that prob­a­bil­ity is high in ev­ery situ­a­tion and save one’s cog­ni­tive re­sources for higher im­pact calcu­la­tions.

• Thanks! I com­pletely agree that “re­ject bets offered to you by hu­mans” is a de­cent heuris­tic that hu­mans seem to use. I also agree that bet-stigma is a large part of the rea­son peo­ple feel they need some­thing other than Bayesi­anism (which treats ev­ery choice as a bet about which available ac­tion is best). Th­ese points (and oth­ers) are cov­ered in the next post.

In this post, I’m ad­dress­ing the ar­gu­ment that there are ra­tio­nal prefer­ences that the Bayesian frame­work can­not, in prin­ci­ple, cap­ture. This ad­dresses a more gen­eral con­cern as to whether Bayesi­anism cap­tures the in­tu­itive ideal of ‘ra­tio­nal­ity’. Here I’m claiming that, at least, the MMEU rule is no counter-ex­am­ple. The next post will con­tain my true re­jec­tion of the MMEU rule in par­tic­u­lar.

• No mat­ter how ob­vi­ous your rea­son­ing may ap­pear to you, there is some­one out there stupid enough to have thought the con­trary. Believe it or not, this se­ries goes a long way to­wards dis­si­pat­ing my pes­simism about the world. My sub­con­scious re­ally be­lieved it is a fact that on av­er­age, na­ture tends to de­stroy our mor­tal am­bi­tions, and that’s why it is dan­ger­ous to Tempt Fate.

I have always known this is a the­olog­i­cal out­look, but I tried to deal with it by avoid­ing thoughts like that rather than mar­shal­ing pos­i­tive ar­gu­ments against it. After read­ing this, I con­sciously un­der­stand, to a sig­nifi­cantly greater de­gree, why it doesn’t ac­tu­ally make sense to gen­er­al­ize those thought pro­cesses for use in rea­son­ing. I like this much bet­ter than just in­tu­itively la­bel­ing them as low sta­tus. Thank you.

• It’s fairly com­mon in pro­gram­ming, par­tic­u­larly, to care not just about the av­er­age case be­hav­ior, but the worst case as well. Taken to an ex­treme, this looks a lot like Caul, but treated as a par­tial but not over­whelming fac­tor, it seems rea­son­able and proper.

For ex­am­ple, imag­ine some al­gorithm which will be used very fre­quently, and for which the dis­tri­bu­tion of in­puts is un­cer­tain. The best av­er­age-case re­sponse time achiev­able is 125 ms, but this al­gorithm has high var­i­ance such that most of the time it will re­spond in 120 ms, but a very small pro­por­tion of the time it will take 20 full sec­onds. Another al­gorithm has av­er­age re­sponse time 150 ms, and will never take longer than 200 ms. Gen­er­ally, the sec­ond al­gorithm is a bet­ter choice; av­er­age-case perfor­mance is im­por­tant, but sac­ri­fic­ing some perfor­mance to re­duce the var­i­ance is worth­while.

Tak­ing this ex­am­ple to ex­tremes seems to pro­duce Caul-like de­ci­sion­mak­ing. I agree that Caul ap­pears in­sane, but I don’t see any way ei­ther that this ex­am­ple is wrong, or that the logic breaks down while tak­ing it to ex­tremes.

• The most ob­vi­ous ex­pla­na­tion for this is that util­ity is not a lin­ear func­tion of re­sponse time: the al­gorithm tak­ing 20 s is very, very bad, and los­ing 25 ms on av­er­age is worth­while to en­sure that this never hap­pens. Con­sider that if the al­gorithm is just do­ing some­thing im­me­di­ately prof­itable with no in­ter­ac­tions with any­thing else (e.g. pro­duc­ing some crytp­tocur­rency), the first al­gorithm is clearly bet­ter (as­sum­ing you are just try­ing to max­i­mize ex­pected profit), since on the rare oc­ca­sions when it takes 20 s, you just have to wait al­most 200 times as long for your unit of profit. This sug­gests that the only rea­son the sec­ond al­gorithm is typ­i­cally preferred is that most pro­grams do have to in­ter­act with other things, and an ex­tremely long re­sponse time will break ev­ery­thing. I don’t think any more con­voluted de­ci­sion the­o­retic rea­son­ing is nec­es­sary to jus­tify this.

• True, but even in cases where it won’t break ev­ery­thing, this is still val­ued. Con­sis­tency is a virtue even if in­con­sis­tency won’t break any­thing. And it clearly breaks down in the ex­treme case where it be­comes Caul, but I can’t come up with a com­pel­ling rea­son why it should break down.

My best guess: The fac­tor that is be­ing val­ued here is the var­i­ance. Low var­i­ance in­creases util­ity gen­er­ally, be­cause pre­dictabil­ity is valuable in en­abling bet­ter ex­pected util­ity calcu­la­tions for other con­nected de­ci­sions. There is no hard limit on how much this can mat­ter rel­a­tive to the av­er­age case, but as the dis­crep­ancy be­tween the av­er­age cases di­verge so that the low-var­i­ance ver­sion be­comes worse than a greater and greater frac­tion of the high-var­i­ance cases, it it re­mains tech­ni­cally ra­tio­nal but its im­plicit prior ap­proaches an in­sane prior such as that of Caul or Perry.

I think this would im­ply that for an un­bounded perfect Bayesian, there is no value to low var­i­ance out­side of non­lin­ear util­ity de­pen­dence, but that for bounded rea­son­ers, there is some cut­off where mak­ing con­ces­sions to pre­dictabil­ity de­spite loss of av­er­age-case util­ity is use­ful on bal­ance.

• Cau­tious Caul is in­ter­est­ing be­cause I ac­tu­ally do ex­pect that my util­ity is non­lin­ear with re­spect to mea­sure. For in­stance, if I got to choose be­tween ei­ther the en­tire uni­verse get­ting de­stroyed with prob­a­bil­ity 12 or half of Everett branches get­ting de­stroyed with prob­a­bil­ity 1, I would much pre­fer the sec­ond one. That said, in prac­tice, I don’t ex­pect to be able to make any use of the dis­tinc­tion be­tween mea­sure in a mul­ti­verse and prob­a­bil­ity.