Knightian Uncertainty: Bayesian Agents and the MMEU rule

Re­cently, I found my­self in a con­ver­sa­tion with some­one ad­vo­cat­ing the use of Knigh­tian un­cer­tainty. We both agreed that there’s no point in singling out some of your un­cer­tainty as “spe­cial” un­less you treat it differ­ently.

I am un­der the im­pres­sion that most ad­vice from ad­vo­cates of Knigh­tian un­cer­tainty can be taken to heart in a Bayesian frame­work, and so I find the con­cept of “Knigh­tian un­cer­tainty” un­com­pel­ling. My friend, who I’m anonymiz­ing as “Sir Percy”, claims that he does treat Knigh­tian un­cer­tainty differ­ently from nor­mal un­cer­tainty, and so he needs to make the dis­tinc­tion. Un­like an as­piring Bayesian rea­soner, who at­tempts to max­i­mize ex­pected value, he max­i­mizes the min­i­mum ex­pected value given his Knigh­tian un­cer­tainty. This is the MMEU rule mo­ti­vated pre­vi­ously.

This sur­prised me: is it pos­si­ble for a ra­tio­nal agent to re­fuse to max­i­mize ex­pected util­ity? My re­flex­ive re­ac­tion was sim­ple:

If you’re a ra­tio­nal agent and you don’t think you’re max­i­miz­ing ex­pected util­ity, then you’ve mis­placed your “util­ity” la­bel.

Sir Percy had a re­sponse ready:

That can’t be. Re­mem­ber Sir Percy’s coin toss. A coin has been tossed, and the event “H” is “the coin came up heads”. Con­sider the fol­low­ing two bets:

  1. Pay 50¢ to be payed $1.10 if H

  2. Pay 50¢ to be payed $1.10 if ¬H

I don’t know whether the coin was bi­ased, I know only that my cre­dence is in the in­ter­val [0.4, 0.6]. When con­sid­er­ing the first bet, I no­tice that the prob­a­bil­ity of H may be 0.4, in which case the bet is ex­pected to lose me 6¢. When con­sid­er­ing the sec­ond bet, I no­tice that the prob­a­bil­ity of H may be 0.6, in which case the bet is ex­pected to lose me 6¢. But when con­sid­er­ing both to­gether, I see that I will win 10¢ with cer­tainty. So I re­ject each bet if pre­sented in­di­vi­d­u­ally, but I will pay up to 10¢ to play the pair.

As you can see, my prefer­ences change un­der ag­glomer­a­tion of bets. It’s not pos­si­ble to view me as a Bayesian rea­soner max­i­miz­ing ex­pected util­ity, be­cause there is no cre­dence you can as­sign to H such that a Bayesian rea­soner shares my prefer­ences. I can’t be max­i­miz­ing ex­pected util­ity, no mat­ter where you put your la­bels.

My re­jec­tion was vague and un­formed at the time. I’ve since fleshed it out, and it will be pre­sented be­low. But be­fore con­tin­u­ing, see if you can spot my ar­gu­ment on your own:

My friend be­lieves that H oc­cured with prob­a­bil­ity in the in­ter­val [.4, .6]. He is un­will­ing to pay 50¢ to be payed $1.10 if H, and he is un­will­ing to pay 50¢ to be payed $1.10 if ¬H, but he is will­ing to pay up to 10¢ for the pair. There is no way to as­sign a cre­dence to the event H such that these ac­tions are tra­di­tion­ally con­sis­tent. Yet, if we al­low that ra­tio­nal agents can in prin­ci­ple have prefer­ences about am­bi­guity, then he is act­ing ‘ra­tio­nally’ in some sense. Is the Bayesian frame­work ca­pa­ble of cap­tur­ing agents with these prefer­ences?

Short an­swer: Yes.

A Bayesian agent can act like this, and Sir Percy did put his util­ity la­bel in the wrong place. In fact, there are two ways that a Bayesian ideal­iza­tion of Sir Percy can ex­hibit Sir Percy’s prefer­ences. In the coin toss game, at least one of two things is hap­pen­ing:

  1. Sir Percy isn’t treat­ing H like an event.

  2. Sir Percy isn’t treat­ing money like util­ity.

I’ll ex­plore both pos­si­bil­ities in turn, but first, let’s build a lit­tle in­tu­ition:

The prefer­ences of a ra­tio­nal agent are not sup­posed to be in­var­i­ant un­der ag­glomer­a­tion of bets. Imag­ine that there are two po­tions, a blue po­tion and a green po­tion. Each po­tion, taken alone, makes you sick. Both po­tions, taken to­gether, give you su­per­pow­ers. It may well be that you re­ject both “bet 1: 100% you drink the blue po­tion” and “bet 2: 100% you drink the green po­tion”, but that you hap­pily pay for the pair.

This is not ir­ra­tional: You could say there is no as­sign­ment of util­ity to the ac­tion “drink the blue po­tion” that makes my ac­tions con­sis­tent, but this is an er­ror. Prefer­ences are not over ac­tions, they are over out­comes. If I take bet 1, I get sick. If I take bet 2, I get sick. If I take them both to­gether, I get su­per­pow­ers.

The fact that I do not honor ag­glomer­a­tion of bets does not mean I am ir­ra­tional. Sir Percy looks ir­ra­tional not be­cause his prefer­ences di­s­obey ag­glomer­a­tion of bets, but only be­cause there’s no cre­dence which Sir Percy can as­sign to H that makes his ac­tions con­sis­tent with ex­pected money max­i­miza­tion.

This means that, if Sir Percy is tra­di­tion­ally ra­tio­nal, then he’s ei­ther not treat­ing H like an event, or he’s not treat­ing money like util­ity. And as it turns out, ideal­ized Bayesian rea­son­ers can dis­play Sir Percy’s prefer­ences. I’ll illus­trate two of them be­low.

An­tag­o­nis­tic ambiguity

Re­mem­ber Sir Percy’s origi­nal mo­ti­va­tion for re­ject­ing both bets in­di­vi­d­u­ally in Sir Percy’s coin toss:

I’m max­i­miz­ing the min­i­mum ex­pected util­ity. Given bet (1), I no­tice that per­haps the prob­a­bil­ity of H is only 40%, in which case the ex­pected util­ity of bet (1) is −6¢, so I re­ject it. Given bet (2), I no­tice that per­haps the prob­a­bil­ity of H is 60%, in which case the ex­pected util­ity of bet (2) is −6¢, so I re­ject that too.”

No­tice how Sir Percy is act­ing here: given each bet, he as­sumes that the coin is weighted in what­ever man­ner is worst for him. Sir Percy’s [0.4, 0.6] cre­dence in­ter­val for H im­plies that he has seen ev­i­dence that con­vinces him the coin isn’t weighted to land H less than 40% of the time, and isn’t weighted to land H more than 60% of the time, but he can’t nar­row it down any fur­ther than that. And when he’s con­sid­er­ing bets, he acts as­sum­ing that the coin is ac­tu­ally weighted in the least con­ve­nient way.

We can de­sign a Bayesian agent that acts like this, but this Bayesian agent (who I’ll call Para­noid Perry) doesn’t treat H like an event. Rather, Perry acts as if the coin’s weight­ing isn’t cho­sen un­til af­ter Perry se­lects a bet. Then, na­ture gets to choose the weight­ing of the coin (within the con­straints set by Perry’s con­fi­dence in­ter­val), and na­ture always chooses what’s worst for Perry.

Let’s say that Perry is a perfectly ra­tio­nal Bayesian agent. If na­ture has a se­lec­tion of coins weighted from 40% heads to 60% heads, and na­ture gets to se­lect one of the coins af­ter Perry se­lects one of the bets, then re­ject­ing each of the bets in­di­vi­d­u­ally but ac­cept­ing both to­gether is pre­cisely how Perry max­i­mizes ex­pected util­ity. The game seems a bit odd un­til you re­al­ize that Perry isn’t treat­ing H as an event: Perry is act­ing as if it’s choice of bets af­fects the weight­ing of the coin.

No­tice that Perry isn’t treat­ing its [0.4, 0.6] cre­dence in­ter­val for H as if it’s nor­mal un­cer­tainty. Perry is treat­ing this un­cer­tainty as if it is in the world: the coin could ac­tu­ally be weighted any­where from 40% H to 60% H, and na­ture gets to choose. It’s un­clear how Perry de­cides which un­cer­tainty is in­ter­nal (sub­ject to nor­mal Bayesian rea­son­ing) and which un­cer­tainty is ex­ter­nal (re­solved ad­ver­sar­i­ally by na­ture), but this dis­tinc­tion gives rise to the differ­ence be­tween “nor­mal” un­cer­tainty and “Knigh­tian” un­cer­tainty. Given any mechanism for differ­en­ti­at­ing be­tween un­cer­tainty about what world Perry is in and vari­abil­ity that gets to be re­solved by na­ture, we can con­struct a Bayesian rea­soner that acts like Perry.

Take, for ex­am­ple, the Ells­berg urn game. Perry has some­how been con­vinced that the pro­cess of se­lect­ing balls from the urn is fair: balls are se­lected by some stochas­tic pro­cess which Na­ture can­not af­fect. How­ever, Perry also things that the urn in use gets to be cho­sen by na­ture af­ter Perry has se­lected a bet.

There are many mechanisms by which Perry could come to this state of knowl­edge. Per­haps the ball-se­lec­tion mechanism was de­signed and ver­ified by Perry, but the urn is se­lected by an un­ver­ified pro­cess that is sup­pos­edly ran­dom but which Perry has rea­son to be­lieve has ac­tu­ally been in­fected by the ne­far­i­ous spirit of the Ad­ver­sary. Given this knowl­edge, Perry knows that if it chooses bet 1b then na­ture will se­lect the urn with no black balls, and if it chooses bet 2a then na­ture will se­lect the urn with­out yel­low balls, so Perry ra­tio­nally prefers bets 1a and 2b.

Perry is averse to am­bi­guity be­cause Perry be­lieves that na­ture gets to re­solve am­bi­guity, and na­ture is work­ing against Perry. This para­noia cap­tures some of the origi­nal mo­ti­va­tion for am­bi­guity aver­sion, but it starts to seem strange un­der close ex­am­i­na­tion. For ex­am­ple, in the coin toss game, Perry acts as if na­ture gets to pick the weight­ing of the coin even af­ter the coin has been tossed. Perry be­lieves that na­ture gets to re­solve am­bi­guity, even if the am­bi­guity lies in the past.

Fur­ther­more, Perry acts as if it is cer­tain that na­ture gets to re­solve am­bi­guity dis­fa­vor­ably, and this can lead to patholog­i­cal be­hav­ior in some edge cases. For ex­am­ple, con­sider Perry rea­son­ing about the un­bal­anced ten­nis game: Perry be­lieves that one of the play­ers is far bet­ter than the other, ca­pa­ble of win­ning 99 games out of 100. But Perry doesn’t know whether An­abel or Zara is the bet­ter player. How might the con­ver­sa­tion go when a bookie ap­proaches Perry offer­ing a bet of 2:1 odds on the player of Perry’s choice?

Hello, Perry, I’m a rich ec­cen­tric bookie who likes giv­ing peo­ple money in the form of strange bets. I’d like to offer you a bet on the un­bal­anced ten­nis game. I know that you know that one of the play­ers is far bet­ter than the other, and I know you have “Knigh­tian un­cer­tainty” about which player is bet­ter. To coun­ter­mand your con­cerns, I’m go­ing to make you a very good deal: I’m will­ing to offer you a bet with 2:1 odds on the player of your choice.

“I’m sorry”, Perry re­sponds, “I can­not take that bet.”

But why not? I un­der­stand that if I came to you offer­ing a bet with 2:1 odds on An­abel, you should re­ject it and then up­date in fa­vor of An­abel be­ing the bet­ter player, un­der the as­sump­tion that I was try­ing to take ad­van­tage of your un­cer­tainty. But I’m offer­ing you a bet with 2:1 odds on the player of your choice! How can you re­ject this?

“Well, you see”, Perry re­sponds, “no mat­ter what choice I make, na­ture will re­solve my am­bi­guity against me. If I place 2:1 odds on Zara win­ning, then na­ture will de­cide that Zara is the worse player, and Zara will loose 99% of the time. But if I in­stead place 2:1 odds on Zara los­ing, then na­ture will de­cide that Zara is the bet­ter player, and Zara will win 99% of the time. Either way, I lose money.

How can that be? Surely, you be­lieve that one player is already bet­ter than the other. Your choice can’t change who is bet­ter; that was de­cided in the past! You’re act­ing like na­ture will retroac­tively make the one that you bet on be worse!

“Yes, pre­cisely”, Perry re­sponds. “See, I have Knigh­tian un­cer­tainty about which is bet­ter, and na­ture gets to re­solve Knigh­tian un­cer­tainty, re­gard­less of causal­ity. No mat­ter which player I pick, she will turn out to be the worse player.”

Real­iz­ing that this may sound crazy, Perry pauses.

“Now, I’m not sure how this hap­pens. Per­haps na­ture has acausal su­per­pow­ers, ac­tu­ally can al­ter the past. Or per­haps you simu­lated me to see who I would pick, and you’re only offer­ing me this bet be­cause you found that I would pick the wrong player. I don’t know how na­ture does it, I do know for a fact that na­ture will re­solve my am­bi­guity an­tag­o­nis­ti­cally. Of this I am cer­tain. And so I’m sorry, but I can’t take your bet.”

And if I offer you 10:1 odds on the player of your choice?

Perry shakes it’s head. “Sorry.”

The bookie de­ject­edly de­parts.

Perry is a Bayesian agent that shares Sir Percy’s prefer­ences, thereby demon­strat­ing that the Bayesian frame­work can cap­ture agents with prefer­ences over am­bi­guity (given some method for dis­t­in­guish­ing am­bi­guity from un­cer­tainty). Any Bayesian agent that be­lieves that there is ac­tu­ally vari­abil­ity in the en­vi­ron­ment which na­ture gets to acausally ad­ver­sar­i­ally re­solve acts pre­cisely as if it is us­ing the MMEU rule.

In fact, though Perry may seem to be rea­son­ing in an odd fash­ion, Perry cap­tures much of the origi­nal mo­ti­va­tion for the MMEU rule. “I can’t get a pre­cise cre­dence”, Sir Percy would say, “I can only get a cre­dence in­ter­val, be­cause re­al­ity ac­tu­ally still has a lit­tle wig­gle room about whether the coin was weighted. So as­sum­ing the worst case, how do I max­i­mize ex­pected util­ity?”

The para­noia of Perry is a lit­tle dis­con­cert­ing, though. Is there per­haps an­other ideal­ized Bayesian agent that cap­tures Sir Perry’s prefer­ences in a less abra­sive man­ner?

Prefer­ring the least con­ve­nient world

Allow me to in­tro­duce an­other Bayesian agent, Cau­tious Caul. Like Perry, Caul dis­t­in­guishes be­tween differ­ent types of un­cer­tainty. The method of dis­tinc­tion is left un­speci­fied, but given any means of differ­en­ti­at­ing be­tween un­cer­tainty and am­bi­guity (“Knigh­tian un­cer­tainty”), we can spec­ify an agent that acts like Caul.

Like Perry, Caul treats am­bi­guity differ­ently form nor­mal un­cer­tainty be­cause Caul be­lieves that am­bi­guity is not in­ter­nal lack of knowl­edge, but rather a fact about the ex­ter­nal world. But whereas Perry thinks that Knigh­tian un­cer­tainty de­notes vari­abil­ity in the world that is re­solved by na­ture, Caul in­stead thinks that am­bi­guity de­notes wor­ld­parts that ac­tu­ally ex­ist.

And Caul is an ex­pected util­ity max­i­mizer (and a perfect Bayesian), but Caul’s prefer­ences are defined ac­cord­ing to the least con­ve­nient world.

When faced with Sir Percy’s coin toss, Caul hon­estly be­lieves that a con­tin­u­ous set of wor­lds ex­ist, each with a coin weighted some­where be­tween 40% H and 60% H, and that in each of these wor­lds there is a sliver of Caul. But Caul’s prefer­ences are defined such that Caul only cares about the sliver in the least con­ve­nient world, the world with the worst odds.

In other words, ev­ery Caul-sliver is will­ing to trade un­limited amounts of ex­pected dol­lars in their own wor­l­part in or­der to in­crease the ex­pected dol­lars of the Caul in the least con­ve­nient wor­ld­part. Thus, Caul re­fuses the first bet in Sir Percy’s coin toss, be­cause con­di­tioned upon Caul tak­ing the first bet, Caul only cares about the wor­ld­part where the coin is weighted 40% H (and so Caul re­fuses the bet). But con­di­tioned upon Caul tak­ing the sec­ond bet, Caul only cares about the wor­ld­part where the coin is weighted 60% H (and so Caul re­fuses the bet). But when offered both bets si­mul­ta­neously, all the Caul slivers in all the wor­ld­parts will gain 10¢, so Caul takes the com­bi­na­tion.

In this case, we see why there is no Bayesian with a sin­gle cre­dence for H that can ex­hibit Caul’s prefer­ences. From Caul’s point of view, H is not a sin­gle event, it is a set of events with vary­ing cre­dence. Any Bayesian at­tempt­ing to as­sign a sin­gle cre­dence to H is ne­glect­ing a cru­cial part of (what Caul sees as) the world’s struc­ture.

Fur­ther­more, Caul also fails to treat the out­comes of the bet (in dol­lars) as util­ity: Caul only cares about the least con­ve­nient world given Caul’s ac­tion, so the util­ity that Caul gets from a bet is de­pen­dent upon Caul’s ac­tion. We can see this when imag­in­ing Caul fac­ing the Ells­berg urn game. In this sce­nario, Caul has ac­tu­ally come to be­lieve that there is a Caul-sliver fac­ing each of the sixty urns. Con­di­tioned upon tak­ing bet 1b, Caul only cares about the Caul-sliver fac­ing the urn with­out any black balls. But con­di­tioned upon tak­ing bet 2a, Caul only cares about the Caul-sliver fac­ing the urn with­out any yel­low balls. Caul’s prefer­ences de­pend upon the ac­tion which Caul chooses.

Caul is an ex­pected util­ity max­i­mizer with prefer­ences only for the wor­ld­part with the worst odds (where wor­ld­parts are de­ter­mined by am­bi­guity and odds are de­ter­mined by un­cer­tainty). Caul may ap­pear to be us­ing the MMEU rule, but call is ac­tu­ally max­i­miz­ing min­i­mum ex­pected dol­lars while max­i­miz­ing ac­tual ex­pected util­ity. This only looks like MMEU if we con­fuse the in­ner util­ity in each wor­ld­part (“dol­lars”) with the outer util­ity of Caul it­self.

To illus­trate, con­sider what hap­pens when Caul en­coun­ters our en­thu­si­as­tic bookie. As with Perry, Caul be­lieves that one of the ten­nis play­ers in the un­bal­anced game could beat the other 99 times out of 100. How­ever, Caul has Knigh­tian un­cer­tainty about whether An­abel or Zara is the su­pe­rior player. This means that Caul acu­tally be­lieves there are two wor­ld­parts, each with a Caul-sliver: one in which An­abel beats Zara 99 times out of 100, and the other in which An­abel loses to Zara 99 times out of 100.

How might Caul’s con­ver­sa­tion with the bookie go?

Ah, Caul! I have an offer for you. You know the un­bal­anced ten­nis game that is about to be played be­tween An­abel and Zara? I’d like to offer you a bet that’s heav­ily in your fa­vor: I’ll give you a bet with 2:1 odds on the player of your choice!

“Sorry”, Caul says, “can’t do that.”

Why not? Look, I’m not try­ing to screw you over here. I just re­ally like giv­ing peo­ple free money, and if I can’t give enough away, I’ll have to an­swer to my board of trustees. I se­ri­ously don’t want to mess with those folks. This re­ally is a bet in your fa­vor! It pays out 2:1. I haven’t simu­lated you, I promise!

“I’m sorry”, Caul replies, “but I can’t. See, if I place the bet on An­abel, then in the world where Zara is the bet­ter player, I ex­pect that ver­sion of Caul to lose money.”

Yes, but in the world where An­abel is the bet­ter player, that ver­sion of Caul wins more than the other ver­sion loses!

“Per­haps, but I only care about the sliver of me with the worst odds. If I place the bet on An­abel, then I care only about the Caul in the world where she’s worse. But if I place the bet on Zara, then I only care about the Caul in the world where she’s bet­ter.” Caul shrugs. “Sorry, there’s just no way I can take the bet with­out mak­ing the least con­ve­nient world worse off, and I only care about what hap­pens in the least con­ve­nient world.”

And so our hap­less bookie de­parts, fran­ti­cally search­ing for some­one to take free money, mut­ter­ing about ir­ra­tional­ity.

Caul cap­tures an­other por­tion of the mo­ti­va­tion for the MMEU rule: Caul only cares about the least con­ve­nient world. Of course, Caul only cares about the least con­ve­nient world given Caul’s nor­mal un­cer­tainty: Caul cares about the Caul-sliver with the worst odds, not with the worst out­come. To see the differ­ence, imag­ine that our bookie offered Caul a bet with 100:1 odds on An­abel. Caul would take this bet, be­cause even in the world with the worst odds the ex­pected Caul-value of this bet is 1¢ (99% of the time Zara wins and Caul loses $1, but 1% of the time An­abel wins and Caul wins $100).

An agent that only cared about the ac­tual worst case (rather than only the wor­ld­part with the worst odds) would re­fuse even this bet, wor­ry­ing about sce­nar­ios where An­abel is a far su­pe­rior player but loses any­way. This would be an agent in which all un­cer­tainty is “Knigh­tian”, and this would turn Caul into a max­i­mizer of min­i­mum ac­tual util­ity (in­stead of min­i­mum ex­pected util­ity), which is wildly im­prac­ti­cal (and of du­bi­ous use­ful­ness). This raises the un­com­fortable ques­tion of how Caul de­cides which un­cer­tain­ties de­note ac­tual wor­ld­parts, and which un­cer­tain­ties de­note “nor­mal” un­cer­tainty with which Caul may gam­ble in the tra­di­tional man­ner.

Perry and Caul show that it’s pos­si­ble for a Bayesian to act ac­cord­ing to Sir Percy’s prefer­ences, but they are only able to do this be­cause they be­lieve that some of their un­cer­tainty is not in­ter­nal (caused by in­com­plete in­for­ma­tion) but rather ex­ter­nal (ac­tu­ally rep­re­sented in the en­vi­ron­ment). Perry be­lieves that some of its un­cer­tainty de­notes places where Na­ture gets to choose how the world works af­ter Perry chooses an ac­tion. Caul be­lieves that some of its un­cer­tainty ac­tu­ally de­notes real wor­ld­parts. In both cases, these agents be­lieve that their am­bi­guity is part of the world, and this is the mechanism by which they can dis­play am­bi­guity aver­sion.

Bayesian am­bi­guity aversion

Perry and Caul help us an­swer our first ques­tion.

Can an agent with prefer­ences about am­bi­guity be ‘ra­tio­nal’ in the Bayesian sense?

To this, we can now an­swer “yes”. Ideal Bayesian agents can ex­hibit the prefer­ences ad­vo­cated by Sir Percy. The next ques­tion is, is this a sane way to act? A ra­tio­nal agent can ex­hibit am­bi­guity aver­sion, and hu­mans seem to ex­hibit am­bi­guity aver­sion… so should we act like this?

Sir Percy ad­vo­cates this de­ci­sion rule. He’s put forth some in­ter­est­ing ar­gu­ments and this post has shown that such a view­point can be ra­tio­nal in a Bayesian sense. But a Bayesian can also have an anti-Lapla­cian prior, so that doesn’t speak to this view­point’s san­ity. Is the MMEU rule a sane way for hu­mans to act?

In short: No, for the same rea­son that Caul and Perry are mad. Their views are con­sis­tent, and their ac­tions are ra­tio­nal (given their pri­ors), but I am re­ally glad that I don’t have their pri­ors. But that’s a point I’ll ex­plore in more depth in the next (and fi­nal) post.