Swimming in Reasons

To a ra­tio­nal­ist, cer­tain phrases smell bad. Rot­ten. A bit fishy. It’s not that they’re ac­tively dan­ger­ous, or that they don’t oc­cur when all is well; but they’re rel­a­tively prone to emerg­ing from cer­tain kinds of thought pro­cesses that have gone bad.

One such phrase is for many rea­sons. For ex­am­ple, many rea­sons all say­ing you should eat some food, or vote for some can­di­date.

To see why, let’s first re­ca­pitu­late how ra­tio­nal up­dat­ing works. Beliefs (in the sense of prob­a­bil­ities for propo­si­tions) ought to bob around in the stream of ev­i­dence as a ran­dom walk with­out trend. When, in con­trast, you can see a be­lief try to swim some­where, right un­der your nose, that’s fishy. (Rot­ten fish don’t re­ally swim, so here the anal­ogy breaks down. Sorry.) As a Less Wrong reader, you’re smarter than a fish. If the fish is go­ing where it’s go­ing in or­der to flee some past er­ror, you can jump ahead of it. If the fish is it­self in er­ror, you can re­fuse to fol­low. The math­e­mat­i­cal for­mu­la­tion of these claims is clearer than the ichthy­olog­i­cal for­mu­la­tion, and can be found un­der con­ser­va­tion of ex­pected ev­i­dence.

More gen­er­ally, ac­cord­ing to the law of iter­ated ex­pec­ta­tions, it’s not just your prob­a­bil­ities that should be free of trends, but your ex­pec­ta­tion of any vari­able. Con­ser­va­tion of ex­pected ev­i­dence is just the spe­cial case where a vari­able can be 1 (if some propo­si­tion is true) or 0 (if it’s false); the ex­pec­ta­tion of such a vari­able is just the prob­a­bil­ity that the propo­si­tion is true.

So let’s look at the case where the vari­able you’re es­ti­mat­ing is an ac­tion’s util­ity. We’ll define a rea­son to take the ac­tion as any info that raises your ex­pec­ta­tion, and the strength of the rea­son as the amount by which it does so. The strength of the next rea­son, con­di­tional on all pre­vi­ous rea­sons, should be dis­tributed with ex­pec­ta­tion zero.

Maybe the dis­tri­bu­tion of rea­sons is sym­met­ri­cal: for ex­am­ple, if some­how you know all rea­sons are equally strong in ab­solute value, rea­sons for and against must be equally com­mon, or they’d cause a pre­dictable trend. Un­der this as­sump­tion, the num­ber of rea­sons in fa­vor will fol­low a bino­mial dis­tri­bu­tion with p=.5. Mostly, the val­ues here will not be too ex­treme, es­pe­cially for large num­bers of rea­sons. When there are ten rea­sons in fa­vor, there are usu­ally at least a few against.

But what if that doesn’t hap­pen? What if ten pieces of info in a row all fa­vor the ac­tion you’re con­sid­er­ing?

One pos­si­bil­ity is you wit­nessed a one in a thou­sand co­in­ci­dence. But let’s not dwell on that. No­body cares about your an­tics in such a tiny slice of pos­si­ble wor­lds.

Another pos­si­bil­ity is the pro­cess gen­er­at­ing new rea­sons con­di­tional on old rea­sons, while un­bi­ased, is not in fact sym­met­ri­cal: it’s skewed. That is to say, it will mostly give a weak rea­son in one di­rec­tion, and in rare cases give a strong rea­son in the other di­rec­tion.

This hap­pens nat­u­rally when you’re con­sid­er­ing many rea­sons for a be­lief, or when there’s some fact rele­vant to an ac­tion that you’re already pretty sure about, but that you’re con­tin­u­ing to in­ves­ti­gate. Fur­ther ev­i­dence will usu­ally bump a high-prob­a­bil­ity be­lief up to­ward 1, be­cause the be­lief is prob­a­bly true; but when it’s bumped down it’s bumped far down. The fact that the sun rose on June 3rd 1978 and the fact that the sun rose on Fe­bru­ary 16th 1860 are both ev­i­dence that the sun will rise in the fu­ture. Each of the many pieces of ev­i­dence like this, taken in­di­vi­d­u­ally, ar­gues weakly against us­ing Aztec-style hu­man sac­ri­fice to pre­vent dawn fail. (If the sun ever failed to rise, that would be a much stronger rea­son the other way, so you’re iter­ated-ex­pec­ta­tions-OK.) If your “many rea­sons” are of this kind, you can stop wor­ry­ing.

Or maybe there’s one com­mon fac­tor that causes many weak rea­sons. Maybe you have a hun­dred le­gi­t­i­mate rea­sons for not hiring some­one as a PR per­son, in­clud­ing that he smashes fur­ni­ture, howls at the moon, and stran­gles kit­tens, all of which make a bad im­pres­sion. If so, you can le­gi­t­i­mately sum­ma­rize your rea­son not to hire him as, “be­cause he’s nuts”. Upon re­al­iz­ing this, you can again stop wor­ry­ing (at least about your own san­ity).

Note that in the pre­vi­ous two cases, if you fail to fully take into ac­count all the im­pli­ca­tions for ex­am­ple, that a per­son in­sane in one way may be in­sane in other ways then it may even seem like there are many rea­sons in one di­rec­tion and none of them are weak.

The last pos­si­bil­ity is the scariest one: you may be one of the fish peo­ple. You may be se­lec­tively look­ing for rea­sons in a par­tic­u­lar di­rec­tion, so you’ll end up in the same place no mat­ter what. Maybe there’s some sort of con­fir­ma­tion bias or halo effect go­ing on.

So in sum, when your brain speaks of “many rea­sons” al­most all go­ing the same way, grab, shake, and stran­gle it. It may just barf up a bet­ter, more com­pressed way of see­ing the world, or con­fess to ul­te­rior mo­tives.

(Thanks to Steve Ray­hawk, Beth Larsen, and Justin Shov­e­lain for com­ments.)

(Clar­ifi­ca­tion in re­sponse to com­ments: I agree that skewed dis­tri­bu­tions are the typ­i­cal case when you’re count­ing pieces of ev­i­dence for a be­lief; the case with the ris­ing sun was meant to cover that, but the post should have been clearer about this point. The sym­met­ri­cal dis­tri­bu­tion as­sump­tion was meant to ap­ply more to, say, many differ­ent good fea­tures of a car, or many differ­ent good con­se­quences of a policy, where the skew doesn’t nat­u­rally oc­cur. Note here the differ­ence be­tween the strength of a rea­son to do some­thing in the sense of how much it bumps up the ex­pected util­ity, and the in­crease in prob­a­bil­ity the rea­son causes for the propo­si­tion that it’s best to do that thing, which gets weaker and weaker the more your es­ti­mate of the util­ity is already higher than the al­ter­na­tives. I said “con­fir­ma­tion bias or halo effect”, but halo effect (prefer­en­tially see­ing good fea­tures of some­thing you already like) is more to the point here than con­fir­ma­tion bias (prefer­en­tially see­ing ev­i­dence for a propo­si­tion you already be­lieve), though many rea­sons in the same di­rec­tion can point to the lat­ter also. I’ve tried to in­cor­po­rate some of this in the post text.)