What is Bayesianism?

This ar­ti­cle is an at­tempt to sum­ma­rize ba­sic ma­te­rial, and thus prob­a­bly won’t have any­thing new for the hard core post­ing crowd. It’d be in­ter­est­ing to know whether you think there’s any­thing es­sen­tial I missed, though.

You’ve prob­a­bly seen the word ‘Bayesian’ used a lot on this site, but may be a bit un­cer­tain of what ex­actly we mean by that. You may have read the in­tu­itive ex­pla­na­tion, but that only seems to ex­plain a cer­tain math for­mula. There’s a wiki en­try about “Bayesian”, but that doesn’t help much. And the LW us­age seems differ­ent from just the “Bayesian and fre­quen­tist statis­tics” thing, too. As far as I can tell, there’s no ar­ti­cle ex­plic­itly defin­ing what’s meant by Bayesi­anism. The core ideas are sprin­kled across a large amount of posts, ‘Bayesian’ has its own tag, but there’s not a sin­gle post that ex­plic­itly comes out to make the con­nec­tions and say “this is Bayesi­anism”. So let me try to offer my defi­ni­tion, which boils Bayesi­anism down to three core tenets.

We’ll start with a brief ex­am­ple, illus­trat­ing Bayes’ the­o­rem. Sup­pose you are a doc­tor, and a pa­tient comes to you, com­plain­ing about a headache. Fur­ther sup­pose that there are two rea­sons for why peo­ple get headaches: they might have a brain tu­mor, or they might have a cold. A brain tu­mor always causes a headache, but ex­ceed­ingly few peo­ple have a brain tu­mor. In con­trast, a headache is rarely a symp­tom for cold, but most peo­ple man­age to catch a cold ev­ery sin­gle year. Given no other in­for­ma­tion, do you think it more likely that the headache is caused by a tu­mor, or by a cold?

If you thought a cold was more likely, well, that was the an­swer I was af­ter. Even if a brain tu­mor caused a headache ev­ery time, and a cold caused a headache only one per cent of the time (say), hav­ing a cold is so much more com­mon that it’s go­ing to cause a lot more headaches than brain tu­mors do. Bayes’ the­o­rem, ba­si­cally, says that if cause A might be the rea­son for symp­tom X, then we have to take into ac­count both the prob­a­bil­ity that A caused X (found, roughly, by mul­ti­ply­ing the fre­quency of A with the chance that A causes X) and the prob­a­bil­ity that any­thing else caused X. (For a thor­ough math­e­mat­i­cal treat­ment of Bayes’ the­o­rem, see Eliezer’s In­tu­itive Ex­pla­na­tion.)

There should be noth­ing sur­pris­ing about that, of course. Sup­pose you’re out­side, and you see a per­son run­ning. They might be run­ning for the sake of ex­er­cise, or they might be run­ning be­cause they’re in a hurry some­where, or they might even be run­ning be­cause it’s cold and they want to stay warm. To figure out which one is the case, you’ll try to con­sider which of the ex­pla­na­tions is true most of­ten, and fits the cir­cum­stances best.

Core tenet 1: Any given ob­ser­va­tion has many differ­ent pos­si­ble causes.

Ac­knowl­edg­ing this, how­ever, leads to a some­what less in­tu­itive re­al­iza­tion. For any given ob­ser­va­tion, how you should in­ter­pret it always de­pends on pre­vi­ous in­for­ma­tion. Sim­ply see­ing that the per­son was run­ning wasn’t enough to tell you that they were in a hurry, or that they were get­ting some ex­er­cise. Or sup­pose you had to choose be­tween two com­pet­ing sci­en­tific the­o­ries about the mo­tion of planets. A the­ory about the laws of physics gov­ern­ing the mo­tion of planets, de­vised by Sir Isaac New­ton, or a the­ory sim­ply stat­ing that the Fly­ing Spaghetti Mon­ster pushes the planets for­wards with His Noodly Ap­pendage. If these both the­o­ries made the same pre­dic­tions, you’d have to de­pend on your prior knowl­edge—your prior, for short—to judge which one was more likely. And even if they didn’t make the same pre­dic­tions, you’d need some prior knowl­edge that told you which of the pre­dic­tions were bet­ter, or that the pre­dic­tions mat­ter in the first place (as op­posed to, say, the­o­ret­i­cal el­e­gance).

Or take the de­bate we had on 9/​11 con­spir­acy the­o­ries. Some peo­ple thought that un­ex­plained and oth­er­wise sus­pi­cious things in the offi­cial ac­count had to mean that it was a gov­ern­ment con­spir­acy. Others con­sid­ered their prior for “the gov­ern­ment is ready to con­duct mas­sively risky op­er­a­tions that kill thou­sands of its own cit­i­zens as a pub­lic­ity stunt”, judged that to be over­whelm­ingly un­likely, and thought it far more prob­a­ble that some­thing else caused the sus­pi­cious things.

Again, this might seem ob­vi­ous. But there are many well-known in­stances in which peo­ple for­get to ap­ply this in­for­ma­tion. Take su­per­nat­u­ral phe­nom­ena: yes, if there were spirits or gods in­fluenc­ing our world, some of the things peo­ple ex­pe­rience would cer­tainly be the kinds of things that su­per­nat­u­ral be­ings cause. But then there are also countless of mun­dane ex­pla­na­tions, from co­in­ci­dences to men­tal di­s­or­ders to an over­ac­tive imag­i­na­tion, that could cause them to per­ceived. Most of the time, pos­tu­lat­ing a su­per­nat­u­ral ex­pla­na­tion shouldn’t even oc­cur to you, be­cause the mun­dane causes already have lots of ev­i­dence in their fa­vor and su­per­nat­u­ral causes have none.

Core tenet 2: How we in­ter­pret any event, and the new in­for­ma­tion we get from any­thing, de­pends on in­for­ma­tion we already had.

Sub-tenet 1: If you ex­pe­rience some­thing that you think could only be caused by cause A, ask your­self “if this cause didn’t ex­ist, would I re­gard­less ex­pect to ex­pe­rience this with equal prob­a­bil­ity?” If the an­swer is “yes”, then it prob­a­bly wasn’t cause A.

This re­al­iza­tion, in turn, leads us to

Core tenet 3: We can use the con­cept of prob­a­bil­ity to mea­sure our sub­jec­tive be­lief in some­thing. Fur­ther­more, we can ap­ply the math­e­mat­i­cal laws re­gard­ing prob­a­bil­ity to choos­ing be­tween differ­ent be­liefs. If we want our be­liefs to be cor­rect, we must do so.

The fact that any­thing can be caused by an in­finite amount of things ex­plains why Bayesi­ans are so strict about the the­o­ries they’ll en­dorse. It isn’t enough that a the­ory ex­plains a phe­nomenon; if it can ex­plain too many things, it isn’t a good the­ory. Re­mem­ber that if you’d ex­pect to ex­pe­rience some­thing even when your sup­posed cause was un­true, then that’s no ev­i­dence for your cause. Like­wise, if a the­ory can ex­plain any­thing you see—if the the­ory al­lowed any pos­si­ble event—then noth­ing you see can be ev­i­dence for the the­ory.

At its heart, Bayesi­anism isn’t any­thing more com­plex than this: a mind­set that takes three core tenets fully into ac­count. Add a sprin­kle of ideal­ism: a perfect Bayesian is some­one who pro­cesses all in­for­ma­tion perfectly, and always ar­rives at the best con­clu­sions that can be drawn from the data. When we talk about Bayesi­anism, that’s the ideal we aim for.

Fully in­ter­nal­ized, that mind­set does tend to color your thought in its own, pe­cu­liar way. Once you re­al­ize that all the be­liefs you have to­day are based—in a mechanis­tic, lawful fash­ion—on the be­liefs you had yes­ter­day, which were based on the be­liefs you had last year, which were based on the be­liefs you had as a child, which were based on the as­sump­tions about the world that were em­bed­ded in your brain while you were grow­ing in your mother’s womb… it does make you ques­tion your be­liefs more. Won­der about whether all of those pre­vi­ous be­liefs re­ally cor­re­sponded max­i­mally to re­al­ity.

And that’s ba­si­cally what this site is for: to help us be­come good Bayesi­ans.