Comparing Utilities

(This is a ba­sic point about util­ity the­ory which many will already be fa­mil­iar with. I draw some non-ob­vi­ous con­clu­sions which may be of in­ter­est to you even if you think you know this from the ti­tle—but the main point is to com­mu­ni­cate the ba­sics. I’m post­ing it to the al­ign­ment fo­rum be­cause I’ve heard mi­s­un­der­stand­ings of this from some in the AI al­ign­ment re­search com­mu­nity.)

I will first give the ba­sic ar­gu­ment that the util­ity quan­tities of differ­ent agents aren’t di­rectly com­pa­rable, and a few im­por­tant con­se­quences of this. I’ll then spend the rest of the post dis­cussing what to do when you need to com­pare util­ity func­tions.

Utilities aren’t com­pa­rable.

Utility isn’t an or­di­nary quan­tity. A util­ity func­tion is a de­vice for ex­press­ing the prefer­ences of an agent.

Sup­pose we have a no­tion of out­come.* We could try to rep­re­sent the agent’s prefer­ences be­tween out­comes as an or­der­ing re­la­tion: if we have out­comes A, B, and C, then one pos­si­ble prefer­ence would be A<B<C.

How­ever, a mere or­der­ing does not tell us how the agent would de­cide be­tween gam­bles, ie, situ­a­tions giv­ing A, B, and C with some prob­a­bil­ity.

With just three out­comes, there is only one thing we need to know: is B closer to A or C, and by how much?

We want to con­struct a util­ity func­tion U() which rep­re­sents the prefer­ences. Let’s say we set U(A)=0 and U(C)=1. Then we can rep­re­sent B=G as U(B)=1/​2. If not, we would look for a differ­ent gam­ble which does equal B, and then set B’s util­ity to the ex­pected value of that gam­ble. By as­sign­ing real-num­bered val­ues to each out­come, we can fully rep­re­sent an agent’s prefer­ences over gam­bles. (As­sum­ing the VNM ax­ioms hold, that is.)

But the ini­tial choices U(A)=0 and U(C)=1 were ar­bi­trary! We could have cho­sen any num­bers so long as U(A)<U(C), re­flect­ing the prefer­ence A<C. In gen­eral, a valid rep­re­sen­ta­tion of our prefer­ences U() can be mod­ified into an equally valid U’() by adding/​sub­tract­ing ar­bi­trary num­bers, or mul­ti­ply­ing/​di­vid­ing by pos­i­tive num­bers.

So it’s just as valid to say some­one’s ex­pected util­ity in a given situ­a­tion is 5 or −40, pro­vided you shift ev­ery­thing else around ap­pro­pri­ately.

Writ­ing to mean that two util­ity func­tions rep­re­sent the same prefer­ences, what we have in gen­eral is: if and only if . (I’ll call the mul­ti­plica­tive con­stant and the ad­di­tive con­stant.)

This means that we can’t di­rectly com­pare the util­ity of two differ­ent agents. No­tions of fair­ness should not di­rectly say “ev­ery­one should have the same ex­pected util­ity”. Utili­tar­ian ethics can­not di­rectly max­i­mize the sum of ev­ery­one’s util­ity. Both of these op­er­a­tions should be thought of as a type er­ror.

Some non-ob­vi­ous con­se­quences.

The game-the­ory term “zero sum” is a mis­nomer. You shouldn’t di­rectly think about the sum of the util­ities.

In mechanism de­sign, ex­change­able util­ity is a use­ful as­sump­tion which is of­ten needed in or­der to get nice re­sults. The idea is that agents can give utils to each other, per­haps to com­pen­sate for un­fair out­comes. This is kind of like as­sum­ing there’s money which can be ex­changed be­tween agents. How­ever, the non-com­pa­ra­bil­ity of util­ity should make this seem re­ally weird. (There are also other dis­analo­gies with money; for ex­am­ple, util­ity is closer to log­a­r­ith­mic in money, not lin­ear.)

This could (should?) also make you sus­pi­cious of talk of “av­er­age util­i­tar­i­anism” and “to­tal util­i­tar­i­anism”. How­ever, be­ware: only one kind of “util­i­tar­i­anism” holds that the term “util­ity” in de­ci­sion the­ory means the same thing as “util­ity” in ethics: namely, prefer­ence util­i­tar­i­anism. Other kinds of util­i­tar­i­anism can dis­t­in­guish be­tween these two types of util­ity. (For ex­am­ple, one can be a he­do­nic util­i­tar­ian with­out think­ing that what ev­ery­one wants is hap­piness, if one isn’t a prefer­ence util­i­tar­ian.)

Similarly, for prefer­ence util­i­tar­i­ans, talk of util­ity mon­sters be­comes ques­tion­able. A util­ity mon­ster is, sup­pos­edly, some­one who gets much more util­ity out of re­sources than ev­ery­one else. For a he­do­nic util­i­tar­ian, it would be some­one who ex­pe­riences much deeper sad­ness and much higher heights of hap­piness. This per­son sup­pos­edly mer­its more re­sources than other peo­ple.

For a prefer­ence util­i­tar­ian, in­com­pa­ra­bil­ity of util­ity means we can’t sim­ply posit such a util­ity mon­ster. It’s mean­ingless a pri­ori to say that one per­son sim­ply has much stronger prefer­ences than an­other (in the util­ity func­tion sense).

All that be­ing said, we can ac­tu­ally com­pare util­ities, sum them, ex­change util­ity be­tween agents, define util­ity mon­sters, and so on. We just need more in­for­ma­tion.

Com­par­ing util­ities.

The in­com­pa­ra­bil­ity of util­ity func­tions doesn’t mean we can’t trade off be­tween the util­ities of differ­ent peo­ple.

I’ve heard the non-com­pa­ra­bil­ity of util­ity func­tions sum­ma­rized as the the­sis that we can’t say any­thing mean­ingful about the rel­a­tive value of one per­son’s suffer­ing vs an­other per­son’s con­ve­nience. Not so! Rather, the point is just that we need more as­sump­tions in or­der to say any­thing. The util­ity func­tions alone aren’t enough.

Pareto-Op­ti­mal­ity: The Min­i­mal Standard

Com­par­ing util­ity func­tions sug­gests putting them all onto one scale, such that we can trade off be­tween them—“this dol­lar does more good for Alice than it does for Bob”. We for­mal­ize this by imag­in­ing that we have to de­cide policy for the whole group of peo­ple we’re con­sid­er­ing (e.g., the whole world). We con­sider a so­cial choice func­tion which would make those de­ci­sions on be­half of ev­ery­one. Sup­pos­ing it is VNM ra­tio­nal, its de­ci­sions must be com­pre­hen­si­ble in terms of a util­ity func­tion, too. So the prob­lem re­duces to com­bin­ing a bunch of in­di­vi­d­ual util­ity func­tions, to get one big one.

So, how do we go about com­bin­ing the prefer­ences of many agents into one?

The first and most im­por­tant con­cept is the pareto im­prove­ment: our so­cial choice func­tion should en­dorse changes which benefit some­one and harm no one. An op­tion which al­lows no such im­prove­ments is said to be Pareto-op­ti­mal.

We might also want to con­sider strict Pareto im­prove­ments: a change which benefits ev­ery­one. (An op­tion which al­lows no strict Pareto im­prove­ments is weakly Pareto-op­ti­mal.) Strict Pareto im­prove­ments can be more rele­vant in a bar­gain­ing con­text, where you need to give ev­ery­one some­thing in or­der to get them on board with a pro­posal—oth­er­wise they may judge the im­prove­ment as un­fairly fa­vor­ing oth­ers. How­ever, in a bar­gain­ing con­text, in­di­vi­d­u­als may re­fuse even a strict Pareto im­prove­ment due to fair­ness con­sid­er­a­tions.

In ei­ther case, a ver­sion of Harsanyi’s util­i­tar­i­anism The­o­rem im­plies that the util­ity of our so­cial choice func­tion can be un­der­stood as some lin­ear com­bi­na­tion of the in­di­vi­d­ual util­ity func­tions.

So, pareto-op­ti­mal so­cial choice func­tions can always be un­der­stood by:

  1. Choos­ing a scale for ev­ery­one’s util­ity func­tion—IE, set the mul­ti­plica­tive con­stant. (If the so­cial choice func­tion is only weakly Pareto op­ti­mal, some of the mul­ti­plica­tive con­stants might turn out to be zero, to­tally can­cel­ling out some­one’s in­volve­ment. Other­wise, they can all be pos­i­tive.)

  2. Ad­ding all of them to­gether.

(Note that the ad­di­tive con­stant doesn’t mat­ter—shift­ing a per­son’s util­ity func­tion up or down doesn’t change what de­ci­sions will be en­dorsed by the sum. How­ever, it will mat­ter for some other ways to com­bine util­ity func­tions.)

This is nice, be­cause we can always com­bine ev­ery­thing lin­early! We just have to set things to the right scale and then sum ev­ery­thing up.

How­ever, it’s far from the end of the story. How do we choose mul­ti­plica­tive con­stants for ev­ery­body?

Var­i­ance Nor­mal­iza­tion: Not Too Ex­ploitable?

We could set the con­stants any way we want… to­tally sub­jec­tive es­ti­mates of the worth of a per­son, draw ran­dom lots, etc. But we do typ­i­cally want to rep­re­sent some no­tion of fair­ness. We said in the be­gin­ning that the prob­lem was, a util­ity func­tion has many equiv­a­lent rep­re­sen­ta­tions . We can ad­dress this as a prob­lem of nor­mal­iza­tion: we want to take a and put it into a canon­i­cal form, get­ting rid of the choice be­tween equiv­a­lent rep­re­sen­ta­tions.

One way of think­ing about this is strat­egy-proof­ness. A util­i­tar­ian col­lec­tive should not be vuln­er­a­ble to mem­bers strate­gi­cally claiming that their prefer­ences are stronger (larger ), or that they should get more be­cause they’re worse off than ev­ery­one (smaller -- al­though, re­mem­ber that we haven’t talked about any setup which ac­tu­ally cares about that, yet).

Warm-Up: Range Normalization

Un­for­tu­nately, some ob­vi­ous ways to nor­mal­ize util­ity func­tions are not go­ing to be strat­egy-proof.

One of the sim­plest nor­mal­iza­tion tech­niques is to squish ev­ery­thing into a speci­fied range, such as [0,1]:

This is analo­gous to range vot­ing: ev­ery­one re­ports their prefer­ences for differ­ent out­comes on a fixed scale, and these all get summed to­gether in or­der to make de­ci­sions.

If you’re an agent in a col­lec­tive which uses range nor­mal­iza­tion, then you may want to strate­gi­cally mis-re­port your prefer­ences. In the ex­am­ple shown, the agent has a big hump around out­comes they like, and a small hump on a sec­ondary “just OK” out­come. The agent might want to get rid of the sec­ond hump, forc­ing the group out­come into the more fa­vored re­gion.

I be­lieve that in the ex­treme, the op­ti­mal strat­egy for range vot­ing is to choose some util­ity thresh­old. Any­thing be­low that thresh­old goes to zero, feign­ing max­i­mal dis­ap­proval of the out­come. Any­thing above the thresh­old goes to one, feign­ing max­i­mal ap­proval. In other words, un­der strate­gic vot­ing, range vot­ing be­comes ap­proval vot­ing (range vot­ing where the only op­tions are zero and one).

If it’s not pos­si­ble to mis-re­port your prefer­ences, then the in­cen­tive be­comes to self-mod­ify to liter­ally have these ex­treme prefer­ences. This could per­haps have a real-life analogue in poli­ti­cal out­rage and black-and-white think­ing. If we use this nor­mal­iza­tion scheme, that’s the clos­est you can get to be­ing a util­ity mon­ster.

Var­i­ance Normalization

We’d like to avoid any in­cen­tive to mis­rep­re­sent/​mod­ify your util­ity func­tion. Is there a way to achieve that?

Owen Cot­ton-Bar­ratt dis­cusses differ­ent nor­mal­iza­tion tech­niques in illu­mi­nat­ing de­tail, and ar­gues for var­i­ance nor­mal­iza­tion: di­vide util­ity func­tions by their var­i­ance, mak­ing the var­i­ance one. (Geo­met­ric rea­sons for nor­mal­iz­ing var­i­ance to ag­gre­gate prefer­ences, O Cot­ton-Bar­ratt, 2013.) Var­i­ance nor­mal­iza­tion is strat­egy-proof un­der the as­sump­tion that ev­ery­one par­ti­ci­pat­ing in an elec­tion shares be­liefs about how prob­a­ble the differ­ent out­comes are! (Note that var­i­ance of util­ity is only well-defined un­der some as­sump­tion about prob­a­bil­ity of out­come.) That’s pretty good. It’s prob­a­bly the best we can get, in terms of strat­egy-proof­ness of vot­ing. Will MacAskill also ar­gues for var­i­ance nor­mal­iza­tion in the con­text of nor­ma­tive un­cer­tainty (Nor­ma­tive Uncer­tainty, Will MacAskill, 2014).

In­tu­itively, var­i­ance nor­mal­iza­tion di­rectly ad­dresses the is­sue we en­coun­tered with range nor­mal­iza­tion: an in­di­vi­d­ual at­tempts to make their prefer­ences “loud” by ex­trem­iz­ing ev­ery­thing to 0 or 1. This in­creases var­i­ance, so, is di­rectly pun­ished by var­i­ance nor­mal­iza­tion.

How­ever, Jame­son Quinn, LessWrong’s res­i­dent vot­ing the­ory ex­pert, has warned me rather strongly about var­i­ance nor­mal­iza­tion.

  1. The as­sump­tion of shared be­liefs about elec­tion out­comes is far from true in prac­tice. Jame­son Quinn tells me that, in fact, the strate­gic vot­ing in­cen­tivized by quadratic vot­ing is par­tic­u­larly bad amongst nor­mal­iza­tion tech­niques.

  2. Strat­egy-proof­ness isn’t, af­ter all, the fi­nal ar­biter of the qual­ity of a vot­ing method. The fi­nal ar­biter should be some­thing like the util­i­tar­ian qual­ity of an elec­tion’s out­come. This ques­tion gets a bit weird and re­cur­sive in the cur­rent con­text, where I’m us­ing elec­tions as an anal­ogy to ask how we should define util­i­tar­ian out­comes. But the point still, to some ex­tent, stands.

I didn’t un­der­stand the full jus­tifi­ca­tion be­hind his point, but I came away think­ing that range nor­mal­iza­tion was prob­a­bly bet­ter in prac­tice. After all, it re­duces to ap­proval vot­ing, which is ac­tu­ally a pretty good form of vot­ing. But if you want to do the best we can with the state of vot­ing the­ory, Jame­son Quinn sug­gested 3-2-1 vot­ing. (I don’t think 3-2-1 vot­ing gives us any nice the­ory about how to com­bine util­ity func­tions, though, so it isn’t so use­ful for our pur­poses.)

Open Ques­tion: Is there a var­i­ant of var­i­ance nor­mal­iza­tion which takes differ­ing be­liefs into ac­count, to achieve strat­egy-proof­ness (IE hon­est re­port­ing of util­ity)?

Any­way, so much for nor­mal­iza­tion tech­niques. Th­ese tech­niques ig­nore the broader con­text. They at­tempt to be fair and even-handed in the way we choose the mul­ti­plica­tive and ad­di­tive con­stants. But we could also ex­plic­itly try to be fair and even-handed in the way we choose be­tween Pareto-op­ti­mal out­comes, as with this next tech­nique.

Nash Bar­gain­ing Solution

It’s im­por­tant to re­mem­ber that the Nash bar­gain­ing solu­tion is a solu­tion to the Nash bar­gain­ing prob­lem, which isn’t quite our prob­lem here. But I’m go­ing to gloss over that. Just imag­ine that we’re set­ting the so­cial choice func­tion through a mas­sive ne­go­ti­a­tion, so that we can ap­ply bar­gain­ing the­ory.

Nash offers a very sim­ple solu­tion, which I’ll get to in a minute. But first, a few words on how this solu­tion is de­rived. Nash pro­vides two seper­ate jus­tifi­ca­tions for his solu­tion. The first is a game-the­o­retic deriva­tion of the solu­tion as an es­pe­cially ro­bust Nash equil­ibrium. I won’t de­tail that here; I quite recom­mend his origi­nal pa­per (The Bar­gain­ing Prob­lem, 1950); but, just keep in mind that there is at least some rea­son to ex­pect self­ishly ra­tio­nal agents to hit upon this par­tic­u­lar solu­tion. The sec­ond, un­re­lated jus­tifi­ca­tion is an ax­io­matic one:

  1. In­var­i­ance to equiv­a­lent util­ity func­tions. This is the same mo­ti­va­tion I gave when dis­cussing nor­mal­iza­tion.

  2. Pareto op­ti­mal­ity. We’ve already dis­cussed this as well.

  3. In­de­pen­dence of Ir­rele­vant Alter­na­tives (IIA). This says that we shouldn’t change the out­come of bar­gain­ing by re­mov­ing op­tions which won’t ul­ti­mately get cho­sen any­way. This isn’t even tech­ni­cally one of the VNM ax­ioms, but it es­sen­tially is—the VNM ax­ioms are posed for bi­nary prefer­ences (a > b). IIA is the as­sump­tion we need to break down multi-choice prefer­ences to bi­nary choices. We can jus­tify IIA with a kind of money pump.

  4. Sym­me­try. This says that the out­come doesn’t de­pend on the or­der of the bar­gain­ers; we don’t pre­fer Player 1 in case of a tie, or any­thing like that.

Nash proved that the only way to meet these four crite­ria is to max­i­mize the product of gains from co­op­er­a­tion. More for­mally, choose the out­come which max­i­mizes:

The here is a “sta­tus quo” out­come. You can think of this as what hap­pens if the bar­gain­ing fails. This is some­times called a “threat point”, since strate­gic play­ers should care­fully set what they do if ne­go­ti­a­tion fails so as to max­i­mize their bar­gain­ing po­si­tion. How­ever, you might also want to rule that out, forc­ing to be a Nash equil­ibrium in the hy­po­thet­i­cal game where there is no bar­gain­ing op­por­tu­nity. As such, is also known as the best al­ter­na­tive to ne­go­ti­ated agree­ment (BATNA), or some­times the “dis­agree­ment point” (since it’s what play­ers get if they can’t agree). We can think of sub­tract­ing out as just a way of ad­just­ing the ad­di­tive con­stant, in which case we re­ally are just max­i­miz­ing the product of util­ities. (The BATNA point is always (0,0) af­ter we sub­tract out things that way.)

The Nash solu­tion differs sig­nifi­cantly from the other solu­tions con­sid­ered so far.

  1. Max­i­mize the product?? Didn’t Harsanyi’s the­o­rem guaran­tee we only need to worry about sums?

  2. This is the first pro­posal where the ad­di­tive con­stants mat­ter. In­deed, now the mul­ti­plica­tive con­stants are the ones that don’t mat­ter!

  3. Why wouldn’t any util­ity-nor­mal­iza­tion ap­proach satisfy those four ax­ioms?

Last ques­tion first: how do nor­mal­iza­tion ap­proaches vi­o­late the Nash ax­ioms?

Well, both range nor­mal­iza­tion and var­i­ance nor­mal­iza­tion vi­o­late IIA! If you re­move one of the pos­si­ble out­comes, the nor­mal­iza­tion may change. This makes the so­cial choice func­tion dis­play in­con­sis­tent prefer­ences across differ­ent sce­nar­ios. (But how bad is that, re­ally?)

As for why we can get away with max­i­miz­ing the product, rather than the sum:

The Pareto-op­ti­mal­ity of Nash’s ap­proach guaran­tees that it can be seen as max­i­miz­ing a lin­ear func­tion of the in­di­vi­d­ual util­ities. So Harsanyi’s the­o­rem is still satis­fied. How­ever, Nash’s solu­tion points to a very spe­cific out­come, which Harsanyi doesn’t do for us.

Imag­ine you and me are try­ing to split a dol­lar. If we can’t agree on how to split it, then we’ll end up de­stroy­ing it (rip­ping it dur­ing a des­per­ate at­tempt to wres­tle it from each other’s hands, ob­vi­ously). Thank­fully, John Nash is stand­ing by, and we each agree to re­spect his judge­ment. No mat­ter which of us claims to value the dol­lar more, Nash will al­lo­cate 50 cents to each of us.

Harsanyi hap­pens to see this ex­change, and ex­plains that Nash has cho­sen a so­cial choice func­tion which nor­mal­ized our util­ity func­tions to be equal to each other. That’s the only way Harsanyi can ex­plain the choice made by Nash—the value of the dol­lar was pre­cisely tied be­tween you and me, so a 50-50 split was as good as any other out­come. Harsanyi’s jus­tifi­ca­tion is in­deed con­sis­tent with the ob­ser­va­tion. But why, then, did Nash choose 50-50 pre­cisely? 49-51 would have had ex­actly the same col­lec­tive util­ity, as would 40-60, or any other split!

Hence, Nash’s prin­ci­ple is far more use­ful than Harsanyi’s, even though Harsanyi can jus­tify any ra­tio­nal out­come ret­ro­spec­tively.

How­ever, Nash does rely some­what on that pesky IIA as­sump­tion, whose im­por­tance is per­haps not so clear. Let’s try get­ting rid of that.

Kalai–Smorodinsky

Although the Nash bar­gain­ing solu­tion is the most fa­mous, there are other pro­posed solu­tions to Nash’s bar­gain­ing prob­lem. I want to men­tion just one more, Kalai-Smorod­in­sky (I’ll call it KS).

KS throws out IIA as ir­rele­vant. After all, the set of al­ter­na­tives will af­fect bar­gain­ing. Even in the Nash solu­tion, the set of al­ter­na­tives may have an in­fluence by chang­ing the BATNA! So per­haps this as­sump­tion isn’t so im­por­tant.

KS in­stead adds a mono­ton­ic­ity as­sump­tion: be­ing in a bet­ter po­si­tion should never make me worse off af­ter bar­gain­ing.

Here’s an illus­tra­tion, due to Daniel Dem­ski, of a case where Nash bar­gain­ing fails mono­ton­ic­ity:

I’m not that sure mono­ton­ic­ity re­ally should be an ax­iom, but it does kind of suck to be in an ap­par­ently bet­ter po­si­tion and end up worse off for it. Maybe we could re­late this to strat­egy-proof­ness? A lit­tle? Not sure about that.

Let’s look at the for­mula for KS bar­gain­ing.

Sup­pose there are a cou­ple of dol­lars on the ground: one which you’ll walk by first, and one which I’ll walk by. If you pick up your dol­lar, you can keep it. If I pick up my dol­lar, I can keep mine. But also, if you don’t pick up yours, then I’ll even­tu­ally walk by it and can pick it up. So we get the fol­low­ing:

(The box is filled in be­cause we can also use mixed strate­gies to get val­ues in­ter­me­di­ate be­tween any pure strate­gies.)

Ob­vi­ously in the real world we just both pick up our dol­lars. But, let’s sup­pose we bar­gain about it, just for fun.

The way KS works is, you look at the max­i­mum one player can get (you can get $1), and the max­i­mum the other player could get (I can get $2). Then, al­though we can’t usu­ally jointly achieve those pay­offs (I can’t get $2 at the same time as you get $1), KS bar­gain­ing in­sists we achieve the same ra­tio (I should get twice as much as you). In this case, that means I get $1.33, while you get $0.66. We can vi­su­al­ize this as draw­ing a bound­ing box around the fea­si­ble solu­tions, and draw­ing a di­ag­o­nal line. Here’s the Nash and KS solu­tions side by side:

As in Daniel’s illus­tra­tions, we can vi­su­al­ize max­i­miz­ing the product as draw­ing the largest hy­per­bola we can that still touches the or­ange shape. (Orange dot­ted line.) This sug­gests that we each get $1; ex­actly the same solu­tion as Nash would give for split­ting $2. (The black dot­ted line illus­trates how we’d con­tinue the fea­si­ble re­gion to rep­re­sent a dol­lar-split­ting game, get­ting the full tri­an­gle rather than a chopped off por­tion.) Nash doesn’t care that one of us can do bet­ter than the other; it just looks for the most equal di­vi­sion of funds pos­si­ble, since that’s how we max­i­mize the product.

KS, on the other hand, cares what the max pos­si­ble is for both of us. It there­fore sug­gests that you give up some of your dol­lar to me.

I sus­pect most read­ers will not find the KS solu­tion to be more in­tu­itively ap­peal­ing?

Note that the KS mono­ton­ic­ity prop­erty does NOT im­ply the de­sir­able-sound­ing prop­erty “if there are more op­por­tu­ni­ties for good out­comes, ev­ery­one gets more or is at least not worse off.” (I men­tion this mainly be­cause I ini­tially mis­in­ter­preted KS’s mono­ton­ic­ity prop­erty this way.) In my dol­lar-col­lect­ing ex­am­ple, KS bar­gain­ing makes you worse off sim­ply be­cause there’s an op­por­tu­nity for me to take your dol­lar if you don’t.

Like Nash bar­gain­ing, KS bar­gain­ing ig­nores mul­ti­plica­tive con­stants on util­ity func­tions, and can be seen as nor­mal­iz­ing ad­di­tive con­stants by treat­ing as (0,0). (Note that, in the illus­tra­tion, I as­sumed is cho­sen as (min­i­mal achiev­able for one player, min­i­mal achiev­able for the other). this need not be the case in gen­eral.)

A pe­cu­liar as­pect of KS bar­gain­ing is that it doesn’t re­ally give us an ob­vi­ous quan­tity to max­i­mize, un­like Nash or Harsanyi. It only de­scribes the op­ti­mal point. This seems far less prac­ti­cal, for re­al­is­tic de­ci­sion-mak­ing.

OK, so, should we use bar­gain­ing solu­tions to com­pare util­ities?

My in­tu­ition is that, be­cause of the need to choose the BATNA point , bar­gain­ing solu­tions end up re­ward­ing de­struc­tive threats in a dis­turb­ing way. For ex­am­ple, sup­pose that we are play­ing the dol­lar-split­ting game again, ex­cept that I can costlessly de­stroy $20 of your money, so now in­volves both the de­struc­tion of the $1, and the de­struc­tion of $20. Nash bar­gain­ing now hands the en­tire dol­lar to me, be­cause you are “up $20” in that deal, so the fairest pos­si­ble out­come is to give me the $1. KS bar­gain­ing splits things up a lit­tle, but I still get most of the dol­lar.

If util­i­tar­i­ans were to trade off util­ities that way in the real world, it would benefit pow­er­ful peo­ple, es­pe­cially those will­ing to ex­ploit their power to make cred­ible threats. If X can take ev­ery­thing away from Y, then Nash bar­gain­ing sees ev­ery­thing Y has as already count­ing to­ward “gains from trade”.

As I men­tioned be­fore, some­times peo­ple try to define BATNAs in a way which ex­cludes these kinds of threats. How­ever, I see this as ripe for strate­gic util­ity-spoofing (IE, ly­ing about your prefer­ences, or self-mod­ify­ing to have more ad­van­ta­geous prefer­ences).

So, this might fa­vor nor­mal­iza­tion ap­proaches.

On the other hand, Nash and KS both do way bet­ter in the split-the-dol­lar game than any nor­mal­iza­tion tech­nique, be­cause they can op­ti­mize for fair­ness of out­come, rather than just fair­ness of mul­ti­plica­tive con­stants cho­sen to com­pare util­ity func­tions with.

Is there any ap­proach which com­bines the ad­van­tages of bar­gain­ing and nor­mal­iza­tion??

An­i­mals, etc.

An es­say on util­ity com­par­i­son would be in­com­plete with­out at least men­tion­ing the prob­lem of an­i­mals, plants, and so on.

  • Op­tion one: some cut­off for “moral pa­tients” is defined, such that a util­i­tar­ian only con­sid­ers prefer­ences of agents who ex­ceed the cut­off.

  • Op­tion two: some more con­tin­u­ous no­tion is se­lected, such that we care more about some or­ganisms than oth­ers.

Op­tion two tends to be more ap­peal­ing to me, de­spite the non-egal­i­tar­ian im­pli­ca­tions (e.g., if an­i­mals differ on this spec­trum, than hu­mans could have some vari­a­tion as well).

As already dis­cussed, bar­gain­ing ap­proaches do seem to have this fea­ture: an­i­mals would tend to get less con­sid­er­a­tion, be­cause they’ve got less “bar­gain­ing power” (they can do less harm to hu­mans than hu­mans can do to them). How­ever, this has a dis­taste­ful might-makes-right fla­vor to it.

This also brings to the fore­front the ques­tion of how we view some­thing as an agent. Some­thing like a plant might have quite de­ter­minis­tic ways of re­act­ing to en­vi­ron­men­tal stim­u­lus. Can we view it as mak­ing choices, and thus, as hav­ing prefer­ences? Per­haps “to some de­gree”—if such a de­gree could be defined, nu­mer­i­cally, it could fac­tor into util­ity com­par­i­sons, giv­ing a for­mal way of valu­ing plants and an­i­mals some­what, but “not too much”.

Altru­is­tic agents.

Another puz­zling case, which I think needs to be han­dled care­fully, is ac­count­ing for the prefer­ences of al­tru­is­tic agents.

Let’s pro­ceed with a sim­plis­tic model where agents have “per­sonal prefer­ences” (prefer­ences which just have to do with them­selves, in some sense) and “cofrences” (co-prefer­ences; prefer­ences hav­ing to do with other agents).

Here’s an agent named Sandy:

Sandy
Per­sonal Prefer­encesCofrences
Candy+.1Alice+.1
Pizza+.2Bob-.2
Rain­bows+10Cathy+.3
Kit­tens-20Den­nis+.4

The cofrences rep­re­sent co­effi­cients on other agent’s util­ity func­tions. Sandy’s prefer­ences are sup­posed to be un­der­stood as a util­ity func­tion rep­re­sent­ing Sandy’s per­sonal prefer­ences, plus a weighted sum of the util­ity func­tions of Alice, Bob, Cathy, and Den­nis. (Note that the weights can, hy­po­thet­i­cally, be nega­tive—for ex­am­ple, screw Bob.)

The first prob­lem is that util­ity func­tions are not com­pa­rable, so we have to say more be­fore we can un­der­stand what “weighted sum” is sup­posed to mean. But sup­pose we’ve cho­sen some util­ity nor­mal­iza­tion tech­nique. There are still other prob­lems.

No­tice that we can’t to­tally define Sandy’s util­ity func­tion un­til we’ve defined Alice’s, Bob’s, Cathy’s, and Den­nis’. But any of those four might have cofrences which in­volve Sandy, as well!

Sup­pose we have Avery and Briar, two lovers who “only care about each other”—their only prefer­ence is a cofrence, which places 1.0 value on the other’s util­ity func­tion. We could as­cribe any val­ues at all to them, so long as they’re both the same!

With some tech­ni­cal as­sump­tions (some­thing along the lines of: your cofrences always sum to less than 1), we can en­sure a unique fixed point, elimi­nat­ing any am­bi­guity from the in­ter­pre­ta­tion of cofrences. How­ever, I’m skep­ti­cal of just tak­ing the fixed point here.

Sup­pose we have five siblings: Pri­mus, Se­cun­dus, Ter­tius, Quar­tus, et Quin­tus. All of them value each other at .1, ex­cept Pri­mus, who val­ues all siblings at .2.

If we sim­ply take the fixed point, Pri­mus is go­ing to get the short end of the stick all the time: be­cause Pri­mus cares about ev­ery­one else more, ev­ery­one else cares about Pri­mus’ per­sonal prefer­ences less than any­one else’s.

Sim­ply put, I don’t think more al­tru­is­tic in­di­vi­d­u­als should be pun­ished! In this setup, the “util­ity mon­ster” is the perfectly self­ish in­di­vi­d­ual. Altru­ists will be scram­bling to help this per­son while the self­ish per­son does noth­ing in re­turn.

A differ­ent way to do things is to in­ter­pret cofrences as in­te­grat­ing only the per­sonal prefer­ences of the other per­son. So Sandy wants to help Alice, Cathy, and Den­nis (and harm Bob), but does not au­to­mat­i­cally ex­tend that to want­ing to help any of their friends (or harm Bob’s friends).

This is a lit­tle weird, but gives us a more in­tu­itive out­come in the case of the five siblings: Pri­mus will more of­ten be vol­un­tar­ily helpful to the other siblings, but the other siblings won’t be prej­u­dice against the per­sonal prefer­ences of Pri­mus when weigh­ing be­tween their var­i­ous siblings.

I re­al­ize al­tru­ism isn’t ex­actly sup­posed to be like a bar­gain struck be­tween self­ish agents. But if I think of util­i­tar­i­anism like a coal­i­tion of all agents, then I don’t want it to pun­ish the (self­ish com­po­nent of) the most al­tru­is­tic mem­bers. It seems like util­i­tar­i­anism should have bet­ter in­cen­tives than that?

(Try to take this sec­tion as more of a prob­lem state­ment and less of a solu­tion. Note that the con­cept of cofrence can in­clude, more gen­er­ally, prefer­ences such as “I want to be bet­ter off than other peo­ple” or “I don’t want my util­ity to be too differ­ent from other peo­ple’s in ei­ther di­rec­tion”.)

Utility mon­sters.

Re­turn­ing to some of the points I raised in the “non-ob­vi­ous con­se­quences” sec­tion—now we can see how “util­ity mon­sters” are/​aren’t a con­cern.

On my anal­y­sis, a util­ity mon­ster is just an agent who, ac­cord­ing to your met­ric for com­par­ing util­ity func­tions, has a very large in­fluence on the so­cial choice func­tion.

This might be a bug, in which case you should re­con­sider how you are com­par­ing util­ities. But, since you’ve hope­fully cho­sen your ap­proach care­fully, it could also not be a bug. In that case, you’d want to bite the bul­let fully, defend­ing the claim that such an agent should re­ceive “dis­pro­por­tionate” con­sid­er­a­tion. Pre­sum­ably this claim could be backed up, on the strength of your ar­gu­ment for the util­ity-com­par­i­son ap­proach.

Aver­age util­i­tar­i­anism vs to­tal util­i­tar­i­anism.

Now that we have given some op­tions for util­ity com­par­i­son, can we use them to make sense of the dis­tinc­tion be­tween av­er­age util­i­tar­i­anism and to­tal util­i­tar­i­anism?

No. Utility com­par­i­son doesn’t re­ally help us there.

The av­er­age vs to­tal de­bate is a de­bate about pop­u­la­tion ethics. Harsanyi’s util­i­tar­i­anism the­o­rem and re­lated ap­proaches let us think about al­tru­is­tic poli­cies for a fixed set of agents. They don’t tell us how to think about a set which changes over time, as new agents come into ex­is­tence.

Allow­ing the set to vary over time like this feels similar to al­low­ing a sin­gle agent to change its util­ity func­tion. There is no rule against this. An agent can pre­fer to have differ­ent prefer­ences than it does. A col­lec­tive of agents can pre­fer to ex­tend its al­tru­ism to new agents who come into ex­is­tence.

How­ever, I see no rea­son why pop­u­la­tion ethics needs to be sim­ple. We can have rel­a­tively com­plex prefer­ences here. So, I don’t find para­doxes such as the Repug­nant Con­clu­sion to be es­pe­cially con­cern­ing. To me there’s just this com­pli­cated ques­tion about what ev­ery­one col­lec­tively wants for the fu­ture.

One of the ba­sic ques­tions about util­i­tar­i­anism shouldn’t be “av­er­age vs to­tal?”. To me, this is a type er­ror. It seems to me, more ba­sic ques­tions for a (prefer­ence) util­i­tar­ian are:

  • How do you com­bine in­di­vi­d­ual prefer­ences into a col­lec­tive util­ity func­tion?

    • How do you com­pare util­ities be­tween peo­ple (and an­i­mals, etc)?

      • Do you care about an “ob­jec­tive” solu­tion to this, or do you see it as a sub­jec­tive as­pect of al­tru­is­tic prefer­ences, which can be set in an un­prin­ci­pled way?

      • Do you range-nor­mal­ize?

      • Do you var­i­ance-nor­mal­ize?

      • Do you care about strat­egy-proof­ness?

      • How do you eval­u­ate the bar­gain­ing fram­ing? Is it rele­vant, or ir­rele­vant?

      • Do you care about Nash’s ax­ioms?

      • Do you care about mono­ton­ic­ity?

      • What dis­t­in­guishes hu­mans from an­i­mals and plants, and how do you use it in util­ity com­par­i­son? In­tel­li­gence? Agen­tic­ness? Power? Bar­gain­ing po­si­tion?

    • How do you han­dle cofrences?

*: Agents need not have a con­cept of out­come, in which case they don’t re­ally have a util­ity func­tion (be­cause util­ity func­tions are func­tions of out­comes). How­ever, this does not sig­nifi­cantly im­pact any of the points made in this post.