Intertheoretic utility comparison

This is pre­sent­ing some old work on com­bin­ing differ­ent pos­si­ble util­ity func­tions, that is worth re­veal­ing to the world.

I’ve writ­ten be­fore about the prob­lem of reach­ing an agree­ment be­tween agents with differ­ent util­ity func­tions. The prob­lem re-ap­pears if you your­self are un­cer­tain be­tween two differ­ent moral the­o­ries.

For ex­am­ple, sup­pose you gave cre­dence to av­er­age util­i­tar­i­anism and cre­dence to to­tal util­i­tar­i­anism. In an oth­er­wise empty uni­verse, you can cre­ate one per­son with util­ity, or a thou­sand with util­ity.

If we naively com­puted the ex­pected util­ity of both ac­tions, we would get for the first choice, and for the sec­ond. It there­fore seems that to­tal util­i­tar­i­anism wins by de­fault, even though it is very un­likely (for you).

But the situ­a­tion can be worse. Sup­pose that there is a third op­tion, which cre­ated ten thou­sand peo­ple with each util­ity. And you have cre­dence on av­er­age util­i­tar­i­anism, cre­dence on to­tal util­i­tar­i­anism, and cre­dence on ex­po­nen­tial util­i­tar­i­anism, where the av­er­age util­ity is mul­ti­plied by two to the power of the pop­u­la­tion. In this case the third op­tion—and the in­cred­ibly un­likely ex­po­nen­tial util­i­tar­i­anism—win out mas­sively.

Nor­mal­is­ing utilities

To pre­vent the large-pop­u­la­tion-lov­ing util­ities from win­ning out by de­fault, it’s clear we need to nor­mal­ise the util­ities in some way be­fore adding them to­gether, similarly to how you nor­mal­ise the util­ities of op­pos­ing agents.

I’ll dis­t­in­guish two meth­ods here: in­di­vi­d­ual nor­mal­i­sa­tions, and col­lec­tive nor­mal­i­sa­tions. For in­di­vi­d­ual nor­mal­i­sa­tions, if you have cre­dences of for util­ities , then is nor­mal­ised into us­ing some pro­ce­dure that is in­de­pen­dent of , , and for . Then the nor­mal­ised util­ities are added to give your to­tal util­ity func­tion of:

In col­lec­tive nor­mal­i­sa­tions, the nor­mal­i­sa­tion of into is al­lowed to de­pend upon the other util­ities and the cre­dences. All Pareto out­comes for the util­ities are equiv­a­lent (mod­ulo re­solv­ing ties) with max­imis­ing such a .

The Nash Bar­gain­ing Equil­ibrium and the Kalai-Smorod­in­sky Bar­gain­ing Solu­tion are both col­lec­tive nor­mal­i­sa­tions; the Mu­tual Worth Bar­gain­ing Solu­tion is an in­di­vi­d­ual nor­mal­i­sa­tion iff the choice of the de­fault point is in­di­vi­d­ual (but do­ing that vi­o­lates the spirit of what that method is sup­posed to achieve).

Note that there are no non-dic­ta­to­rial Pareto nor­mal­i­sa­tions, whether in­di­vi­d­ual or col­lec­tive, that are in­de­pen­dent of ir­rele­vant al­ter­na­tives, or that are im­mune to ly­ing.

In­di­vi­d­ual normalisations

Here I’ll pre­sent the work that I did with Owen Cot­ton-Bar­ratt, Toby Ord, and Will MacAskill, in or­der to try and come up with a prin­ci­pled way of do­ing in­di­vi­d­ual nor­mal­i­sa­tions. In a cer­tain sense, this work failed: we didn’t find any nor­mal­i­sa­tions that were clearly su­pe­rior in ev­ery way to oth­ers. But we did find a lot about the prop­er­ties of the differ­ent nor­mal­i­sa­tions; one in­ter­est­ing thing is that the dumb­est nor­mal­sa­tion—the zero-one, or min-max—has sur­pris­ingly good prop­er­ties.

Let be the op­tion set for the agent: the choices that it can make (in our full treat­ment, we con­sid­ered a larger set , the nor­mal­i­sa­tion set, but this won’t be needed here).

For the pur­pose of this post, will be equal to , the set of de­ter­minis­tic poli­cies the agent can fol­low; this feels like a nat­u­ral choice, as it’s what the agent re­ally has con­trol over.

For any and , there is the ex­pected util­ity of con­di­tional on the agent fol­low­ing policy ; this will be des­ig­nated by .

We may have a prob­a­bil­ity dis­tri­bu­tion over (maybe defined by the com­plex­ity of the policy?). If we don’t have such a nor­mal­i­sa­tion, and the set of de­ter­minis­tic poli­cies is finite, then we can set to be the uniform dis­tri­bu­tion.

Then, given , each be­comes a real-val­ued ran­dom vari­able, tak­ing value with prob­a­bil­ity . We’ll nor­mal­ise these by nor­mal­is­ing the prop­er­ties of this ran­dom vari­able.

First of all, let’s ex­clude any that are con­stant on all of ; these util­ities can­not be changed, in ex­pec­ta­tion, by the agent’s poli­cies, so should make no differ­ence. Then each , seen as a ran­dom vari­able, has the fol­low­ing prop­er­ties:

  • Max­i­mum: .

  • Min­i­mum: .

  • Mean: .

  • Var­i­ance: .

  • Mean differ­ence: .

There are five nat­u­ral nor­mal­i­sa­tion meth­ods that emerge from these prop­er­ties. The first and most triv­ial is the min-max or zero-one nor­mal­i­sa­tion: scale and trans­late so that takes the value and takes the value (note that the trans­la­tion doesn’t change the de­sired policy when sum­ming util­ities, so what is ac­tu­ally re­quired is to scale so that ).

The sec­ond no­ma­l­i­sa­tion, the mean-max, in­volves set­ting ; by sym­me­try, the min-mean nor­mal­i­sa­tion in­volves set­ting

Fi­nally, the last two nor­mal­i­sa­tions in­volve set­ting ei­ther the var­i­ance, or the mean differ­ence, to .

Mean­ing of the normalisations

What do these nor­mal­i­sa­tions mean? Well, min-max is a nor­mal­i­sa­tion that cares about the differ­ence be­tween perfect utopia and perfect dystopia: be­tween the best pos­si­ble and the worst pos­si­ble ex­pected out­come. Con­cep­tu­ally, this seems prob­le­matic—it’s not clear why the dystopia mat­ters, with seems like some­thing that opens the util­ity up to ex­tor­tion—but, as we’ll see, the min-max nor­mal­i­sa­tion has the best for­mal prop­er­ties.

The mean-max is the nor­mal­i­sa­tion that most ap­peals to me; the mean is the ex­pected value of ran­dom policy, while the max is the ex­pected out­come of the best policy. In a sense, that’s the job of an agent with a sin­gle util­ity func­tion: to move the out­come from ran­dom to best. Thus the max has a mean­ing that the min, for in­stance, lacks.

For this rea­son, I don’t see the min-mean nor­mal­i­sa­tion as be­ing any­thing mean­ingful; it’s the differ­ence be­tween com­plete dis­aster and a ran­dom policy.

I don’t fully grasp the mean­ing of the var­i­ance nor­mal­i­sa­tion; Owen Cot­ton-Bar­ratt did the most work on it, and showed that, in a cer­tain sense, it was re­sis­tant to ly­ing/​strate­gic dis­tor­tion in cer­tain cir­cum­stances, if a given util­ity didn’t ‘know’ what the other util­ities would be. But I didn’t fully un­der­stand this point. So bear in mind that this nor­mal­i­sa­tion has pos­i­tive prop­er­ties that aren’t made clear in this post.

Fi­nally, the mean differ­ence nor­mal­i­sa­tion con­trols the spread be­tween the util­ities of the differ­ent poli­cies, in a lin­ear way that may seem to be more nat­u­ral than the var­i­ance.

Prop­er­ties of the normalisation

So, which nor­mal­i­sa­tion is best? Here’s were we look at the prop­er­ties of the nor­mal­i­sa­tions (they will be sum­marised in a table at the end). As we’ve seen, in­de­pen­dence of ir­rele­vant al­ter­na­tives always fails, and there can always be an in­cen­tive for a util­ity to “lie” (as in, there are , , , , and , such that would have a higher ex­pected util­ity un­der the fi­nal if it was re­placed with ).

What other prop­er­ties do all the nor­mal­i­sa­tions share? Well, since they nor­mal­ise in­de­pen­dently, is con­tin­u­ous in . And be­cause the min­i­mum, max­i­mum, var­i­ance, etc… are con­tin­u­ous in and in , then is also con­tin­u­ous in that in­for­ma­tion.

In con­trast, the best policy of is not typ­i­cally con­tin­u­ous in the data. Imag­ine that there are two util­ities and two poli­cies: and . Then for , is the op­ti­mal policy (for all the above nor­mal­i­sa­tions for uniform ), while for , is op­ti­mal.

Ok, that’s enough of prop­er­ties that all meth­ods share; what about ones they don’t?

First of all, we can look at the nega­tion sym­me­try be­tween and . Min-max, var­i­ance, and mean differ­ence all have the same nor­mal­i­sa­tion for and ; mean-max and min-mean do not, since the mean can be closer to the min than that max (or vice versa).

Then we can con­sider what hap­pens when some poli­cies and are clones of each other: imag­ine that for all , . Then what hap­pens if we re­move the re­dun­dant and nor­mal­ise on ? Well, it’s clear that the max­i­mum or min­i­mum value of can­not change (since if was a max­i­mum/​min­i­mum, then so is , which re­mains), so the min-max nor­mal­i­sa­tion is un­af­fected.

All the other nor­mal­i­sa­tions change, though. This can be seen in the ex­am­ple , , with , , , , and ; in terms of sets of ex­pected util­ities in terms of poli­cies, has while has . Then for uniform , all other nor­mal­i­sa­tion meth­ods change if we re­move which is iden­ti­cal to for both util­ities.

Thus all the other nor­mal­i­sa­tion change when we add (or re­move) clones of ex­ist­ing poli­cies.

Fi­nally, we can con­sider what hap­pens if we are in one of sev­eral wor­lds, and the poli­cies/​util­ities are the iden­ti­cal in some of these wor­lds. This should be treated the same as if those iden­ti­cal wor­lds were all one.

So, imag­ine that we are in one of three wor­lds: , , and , with prob­a­bil­ities , , and , re­spec­tively. Be­fore tak­ing any ac­tions, the agent will dis­cover which world it is in. Thus, if is the set of poli­cies in , the com­plete set of poli­cies is .

The wor­lds and are, how­ever, in­dis­t­in­guish­able for all util­ities in . Thus we can iden­tify , with for all . Then a nor­mal­i­sa­tion method com­bines in­dis­t­in­guish­able choices prop­erty if the nor­mal­i­sa­tion is the same in world and . Then:

  • Min-max, mean-max, and min-mean com­bine in­dis­t­in­guish­able choices. Var­i­ance and mean differ­ence nor­mal­i­sa­tions do not.

Proof (sketch): Let be the ran­dom vari­able that is on un­der the as­sump­tion that is the true un­der­ly­ing world. Then on , be­haves like the ran­dom vari­able . (this means that has prob­a­bil­ity of be­ing , not that it adds ran­dom vari­ables to­gether). Mean, max, and min all have the prop­erty that ; var­i­ance and mean differ­ence, on the other hand, do not.

Sum­mary of properties

In my view, it is a big worry that the var­i­ance and mean differ­ence nor­mal­i­sa­tions fail to com­bine in­dis­t­in­guish­able choices. World and could be strictly iden­ti­cal, ex­cept for some ir­rele­vant in­for­ma­tion that all util­ity func­tions agree is ir­rele­vant. We have to worry about whether the light from a dis­tant star is slightly red­der or slightly bluer than ex­pected; what colour ink was used in a pro­posal; the height of the next an­i­mal we see, and so on.

This means that we can­not di­vide the uni­verse into rele­vant and ir­rele­vant vari­ables, and fo­cus solely on the first.

In table form, the var­i­ous prop­er­ties are:


As can be seen, the min-max method, sim­plis­tic though it is, has all the pos­si­ble nice prop­er­ties.