Extreme risks: when not to use expected utility

Would you pre­fer a 50% chance of gain­ing €10, one chance in a mil­lion off gain­ing €5 mil­lion, or a guaran­teed €5? The stan­dard po­si­tion on Less Wrong is that the an­swer de­pends solely on the differ­ence be­tween cash and util­ity. If your util­ity scales less-than-lin­early with money, you are risk averse and should choose the last op­tion; if it scales more-than-lin­early, you are risk-lov­ing and should choose the sec­ond one. If we re­placed €’s with utils in the ex­am­ple above, then it would sim­ply be ir­ra­tional to pre­fer one op­tion over the oth­ers.

There are math­e­mat­i­cal proofs of that re­sult, but there are also strong in­tu­itive ar­gu­ments for it. What’s the best way of see­ing this? Imag­ine that X1 and X2 were two prob­a­bil­ity dis­tri­bu­tions, with mean u1 and u2 and var­i­ances v1 and v2. If the two dis­tri­bu­tions are in­de­pen­dent, then the sum X1 + X2 has mean u1 + u2, and var­i­ance v1 + v2.

Now if we mul­ti­ply the re­turns of any dis­tri­bu­tion by a con­stant r, the mean scales by r and var­i­ance scales by r2. Con­se­quently if we have n prob­a­bil­ity dis­tri­bu­tions X1, X2, … , Xn rep­re­sent­ing n equally ex­pen­sive in­vest­ments, the ex­pected av­er­age re­turn is (Σni=1 ui)/​n, while the var­i­ance of this av­er­age is (Σni=1 vi)/​n2. If the vn are bounded, then once we make n large enough, that var­i­ance must tend to zero. So if you have many in­vest­ments, your av­er­aged ac­tual re­turns will be, with high prob­a­bil­ity, very close to your ex­pected re­turns.

Thus there is no bet­ter strat­egy than to always fol­low ex­pected util­ity. There is no such thing as sen­si­ble risk-aver­sion un­der these con­di­tions, as there is no ac­tual risk: you ex­pect your re­turns to be your ex­pected re­turns. Even if you your­self do not have enough in­vest­ment op­por­tu­ni­ties to smooth out the un­cer­tainty in this way, you could always ag­gre­gate your own money with oth­ers, through in­surance or in­dex funds, and achieve the same re­sult. Buy­ing a triple-rol­lover lot­tery ticket may be un­wise; but be­ing part of a con­sor­tium that buys up ev­ery ticket for a triple rol­lover lot­tery is just a dull, safe in­vest­ment. If you have al­tru­is­tic prefer­ences, you can even ag­gre­gate re­sults across the planet sim­ply by en­courag­ing more peo­ple to fol­low ex­pected re­turns. So, case closed it seems; de­part­ing from ex­pected re­turns is ir­ra­tional.

But the devil’s de­tail is the con­di­tion ‘once we make n large enough’. Be­cause there are risk dis­tri­bu­tions so skewed that no-one will ever be con­fronted with enough of them to re­duce the var­i­ance to man­age­able lev­els. Ex­treme risks to hu­man­ity are an ex­am­ple; kil­ler as­ter­oids, rogue stars go­ing su­per­nova, un­friendly AI, nu­clear war: even to­tal­ling all these risks to­gether, throw­ing in a few more ex­otic ones, and gen­er­ously adding ev­ery sin­gle other de­ci­sion of our ex­is­tence, we are nowhere near a neat prob­a­bil­ity dis­tri­bu­tion tightly bunched around its mean.

To con­sider an analo­gous situ­a­tion, imag­ine hav­ing to choose be­tween a pro­ject that gave one util to each per­son on the planet, and one that handed slightly over twelve billion utils to a ran­domly cho­sen hu­man and took away one util from ev­ery­one else. If there were trillions of such pro­jects, then it wouldn’t mat­ter what op­tion you chose. But if you only had one shot, it would be pe­cu­liar to ar­gue that there are no ra­tio­nal grounds to pre­fer one over the other, sim­ply be­cause the trillion-iter­ated ver­sions are iden­ti­cal. In the same way, our de­ci­sion when faced with a sin­gle planet-de­stroy­ing event should not be con­strained by the be­havi­our of a hy­po­thet­i­cal be­ing who con­fronts such events trillions of times over.

So where does this leave us? The in­de­pen­dence ax­iom of the von Neu­mann-Mor­gen­stern util­ity for­mal­ism should be ditched, as it im­plies that large var­i­ance dis­tri­bu­tions are iden­ti­cal to sums of low var­i­ance dis­tri­bu­tions. This ax­iom should be re­placed by a weaker ver­sion which re­pro­duces ex­pected util­ity in the limit­ing case of many dis­tri­bu­tions. Since there is no sin­gle ra­tio­nal path available, we need to fill the gap with other ax­ioms – val­ues – that re­flect our gen­uine tol­er­ance to­wards ex­treme risk. As when we first dis­cov­ered prob­a­bil­ity dis­tri­bu­tions in child­hood, we may need to pay at­ten­tion to me­di­ans, modes, var­i­ances, skew­ness, kur­to­sis or the over­all shapes of the dis­tri­bu­tions. Pas­cal’s mug­ger and his whole fam­ily can be con­fronted head-on rather than hop­ing the prob­a­bil­ities neatly can­cel out.

In these ex­treme cases, ex­clu­sively fol­low­ing the ex­pected value is an ar­bi­trary de­ci­sion rather than a log­i­cal ne­ces­sity.