RSS

Bayes’ The­o­rem /​ Bayesianism

Bayes’ The­o­rem (also known as Bayes’ Law) is a law of prob­a­bil­ity that de­scribes the proper way to in­cor­po­rate new ev­i­dence into prior prob­a­bil­ities to form an up­dated prob­a­bil­ity es­ti­mate. It is com­monly re­garded as the foun­da­tion of con­sis­tent ra­tio­nal rea­son­ing un­der un­cer­tainty. Bayes The­o­rem is named af­ter Rev­erend Thomas Bayes who proved the the­o­rem in 1763. Bayesi­anism is the broader philos­o­phy in­spired by the the­o­rem. The core claim be­hind all va­ri­eties of Bayesi­anism is that prob­a­bil­ities are sub­jec­tive de­grees of be­lief—of­ten op­er­a­tional­ized as will­ing­ness to bet.

This stands in con­trast to other in­ter­pre­ta­tions of prob­a­bil­ity, which at­tempt greater ob­jec­tivity. The fre­quen­tist in­ter­pre­ta­tion of prob­a­bil­ity has a fo­cus on re­peat­able ex­per­i­ments; prob­a­bil­ities are the limit­ing fre­quency of an event if you performed the ex­per­i­ment an in­finite num­ber of times.

Another con­tender is the propen­sity in­ter­pre­ta­tion, which grounds prob­a­bil­ity in the propen­sity for things to hap­pen. A perfectly bal­anced 6-sided die would have a 16 propen­sity to land on each side. A propen­sity the­o­rist sees this as a ba­sic fact about dice not de­rived from in­finite se­quences of ex­per­i­ments or sub­jec­tive view­points.

Note how both of these al­ter­na­tive in­ter­pre­ta­tions ground the mean­ing of prob­a­bil­ity in an ex­ter­nal ob­jec­tive fact which can­not be di­rectly ac­cessed.

As a con­se­quence of the sub­jec­tive in­ter­pre­ta­tion of prob­a­bil­ity the­ory, Bayesi­ans are more in­clined to ap­ply Bayes’ The­o­rem in prac­ti­cal statis­ti­cal in­fer­ence. The pri­mary ex­am­ple of this is statis­ti­cal hy­poth­e­sis test­ing. Fre­quen­tists take the ap­pli­ca­tion of Bayes’ The­o­rem to be in­ap­pro­pri­ate, be­cause “the prob­a­bil­ity of a hy­poth­e­sis” is mean­ingless: a hy­poth­e­sis is ei­ther true or false; you can­not define a re­peated ex­per­i­ment in which it is some­times true and some­times false, so you can­not as­sign it an in­ter­me­di­ate prob­a­bil­ity.

Bayes’ the­o­rem com­monly takes the form:

where A is the propo­si­tion of in­ter­est, B is the ob­served ev­i­dence, P(A) and P(B) are prior prob­a­bil­ities, and P(A|B) is the pos­te­rior prob­a­bil­ity of A.

With the pos­te­rior odds, the prior odds and the like­li­hood ra­tio writ­ten ex­plic­itly, the the­o­rem reads:

Vi­su­al­i­sa­tion of Bayes’ Rule

An In­tu­itive Ex­pla­na­tion of Bayes’s Theorem

Eliezer Yudkowsky
1 Jan 2003 20:00 UTC
28 points
0 comments25 min readLW link

A Tech­ni­cal Ex­pla­na­tion of Tech­ni­cal Explanation

Eliezer Yudkowsky
1 Jan 2005 8:00 UTC
45 points
3 comments57 min readLW link

Bayes’ The­o­rem Illus­trated (My Way)

komponisto
3 Jun 2010 4:40 UTC
137 points
195 comments9 min readLW link

It’s Bayes All The Way Up

Scott Alexander
12 Sep 2016 13:35 UTC
20 points
0 comments12 min readLW link

The Equa­tion of Knowledge

Lê Nguyên Hoang
7 Jul 2020 16:09 UTC
59 points
3 comments7 min readLW link

Bayes for Schizophren­ics: Rea­son­ing in Delu­sional Disorders

Scott Alexander
13 Aug 2012 19:22 UTC
96 points
155 comments11 min readLW link

What is Bayesi­anism?

Kaj_Sotala
26 Feb 2010 7:43 UTC
87 points
217 comments4 min readLW link

Learn Bayes Nets!

abramdemski
27 Mar 2018 22:00 UTC
85 points
6 comments2 min readLW link

Bayes’ Law is About Mul­ti­ple Hy­poth­e­sis Testing

abramdemski
4 May 2018 5:31 UTC
83 points
5 comments5 min readLW link

Bayesian examination

Lê Nguyên Hoang
9 Dec 2019 19:50 UTC
86 points
54 comments5 min readLW link

Bayesian Prob­a­bil­ity is for things that are Space-like Separated from You

Scott Garrabrant
10 Jul 2018 23:47 UTC
79 points
22 comments2 min readLW link

What Bayesi­anism taught me

Tyrrell_McAllister
12 Aug 2013 6:59 UTC
70 points
203 comments3 min readLW link

Kal­man Filter for Bayesians

SatvikBeri
22 Oct 2018 17:06 UTC
59 points
6 comments3 min readLW link

Bayesi­anism for Humans

ChrisHallquist
29 Oct 2013 23:54 UTC
57 points
37 comments3 min readLW link

Against strong bayesianism

Richard_Ngo
30 Apr 2020 10:48 UTC
49 points
63 comments6 min readLW link

Bayes’ rule =/​= Bayesian inference

neq1
16 Sep 2010 6:34 UTC
39 points
70 comments2 min readLW link

Bayes Rule Applied

G Gordon Worley III
16 Feb 2018 18:30 UTC
12 points
0 comments1 min readLW link
(towardsdatascience.com)

[Question] al­ter­na­tive his­tory: what if Bayes rule had never been dis­cov­ered?

Yoav Ravid
11 May 2019 7:29 UTC
7 points
3 comments1 min readLW link

nos­talge­braist—bayes: a kinda-sorta masterpost

Kaj_Sotala
4 Sep 2018 11:08 UTC
24 points
19 comments1 min readLW link
(nostalgebraist.tumblr.com)

Fal­la­cies as weak Bayesian evidence

Kaj_Sotala
18 Mar 2012 3:53 UTC
58 points
41 comments10 min readLW link

An­thropic rea­son­ing isn’t magic

Stuart_Armstrong
1 Nov 2017 8:57 UTC
36 points
3 comments2 min readLW link

Kelly bettors

DanielFilan
13 Nov 2018 0:40 UTC
23 points
3 comments10 min readLW link
(danielfilan.com)

Yes, Virginia, You Can Be 99.99% (Or More!) Cer­tain That 53 Is Prime

ChrisHallquist
7 Nov 2013 7:45 UTC
52 points
69 comments6 min readLW link

A His­tory of Bayes’ Theorem

lukeprog
29 Aug 2011 7:04 UTC
57 points
87 comments15 min readLW link

Rad­i­cal Prob­a­bil­ism [Tran­script]

26 Jun 2020 22:14 UTC
45 points
11 comments6 min readLW link

Mini-re­view: ‘Prov­ing His­tory: Bayes’ The­o­rem and the Quest for the His­tor­i­cal Je­sus’

lukeprog
1 Feb 2012 19:20 UTC
22 points
34 comments1 min readLW link

Search­ing for Bayes-Structure

Eliezer Yudkowsky
28 Feb 2008 22:01 UTC
37 points
47 comments5 min readLW link

How to Mea­sure Anything

lukeprog
7 Aug 2013 4:05 UTC
71 points
51 comments22 min readLW link

Refer­ences & Re­sources for LessWrong

XiXiDu
10 Oct 2010 14:54 UTC
123 points
106 comments20 min readLW link

Bayes Questions

Bucky
7 Nov 2018 16:54 UTC
22 points
13 comments2 min readLW link

[Question] What are prin­ci­pled ways for pe­nal­is­ing com­plex­ity in prac­tice?

Bucky
27 Jun 2019 7:28 UTC
42 points
12 comments1 min readLW link

Prob­a­bil­ity is in the Mind

Eliezer Yudkowsky
12 Mar 2008 4:08 UTC
87 points
191 comments6 min readLW link

Bayesian Evolv­ing-to-Extinction

abramdemski
14 Feb 2020 23:55 UTC
39 points
13 comments5 min readLW link

Non-com­mu­ni­ca­ble Evidence

adamzerner
17 Nov 2015 3:46 UTC
10 points
49 comments2 min readLW link

Qual­i­ta­tively Confused

Eliezer Yudkowsky
14 Mar 2008 17:01 UTC
38 points
82 comments4 min readLW link

My Bayesian Enlightenment

Eliezer Yudkowsky
5 Oct 2008 16:45 UTC
34 points
64 comments7 min readLW link

Beau­tiful Probability

Eliezer Yudkowsky
14 Jan 2008 7:19 UTC
59 points
113 comments6 min readLW link

De­co­her­ence is Falsifi­able and Testable

Eliezer Yudkowsky
7 May 2008 7:54 UTC
33 points
35 comments9 min readLW link

Prob­a­bil­ity Space & Au­mann Agreement

Wei_Dai
10 Dec 2009 21:57 UTC
44 points
74 comments5 min readLW link

Au­mann’s Agree­ment Revisited

agilecaveman
27 Aug 2018 6:21 UTC
4 points
1 comment7 min readLW link

The Me­chan­ics of Disagreement

Eliezer Yudkowsky
10 Dec 2008 14:01 UTC
8 points
26 comments4 min readLW link

The Joys of Con­ju­gate Priors

TCB
21 May 2011 2:41 UTC
43 points
24 comments5 min readLW link

Bayesi­anism for hu­mans: “prob­a­ble enough”

BT_Uytya
2 Sep 2014 21:44 UTC
38 points
7 comments4 min readLW link

Nav­i­gat­ing dis­agree­ment: How to keep your eye on the ev­i­dence

AnnaSalamon
24 Apr 2010 22:47 UTC
37 points
73 comments6 min readLW link

If It’s Worth Do­ing, It’s Worth Do­ing With Made-Up Statistics

Scott Alexander
3 Sep 2017 20:56 UTC
36 points
0 comments5 min readLW link

Beyond Bayesi­ans and Frequentists

jsteinhardt
31 Oct 2012 7:03 UTC
36 points
51 comments11 min readLW link

Born as the sev­enth month dies …

Rudi C
10 Jul 2020 15:07 UTC
6 points
8 comments2 min readLW link

[Question] Can Bayes the­o­rem rep­re­sent in­finite con­fu­sion?

Yoav Ravid
22 Mar 2019 18:02 UTC
4 points
13 comments1 min readLW link

Why We Can’t Take Ex­pected Value Es­ti­mates Liter­ally (Even When They’re Un­bi­ased)

HoldenKarnofsky
18 Aug 2011 23:34 UTC
95 points
252 comments17 min readLW link

Fre­quen­tist Statis­tics are Fre­quently Subjective

Eliezer Yudkowsky
4 Dec 2009 20:22 UTC
71 points
82 comments8 min readLW link

Dou­ble Illu­sion of Transparency

Eliezer Yudkowsky
24 Oct 2007 23:06 UTC
66 points
32 comments3 min readLW link

Against NHST

gwern
21 Dec 2012 4:45 UTC
63 points
64 comments4 min readLW link

In­ter­pre­ta­tions of “prob­a­bil­ity”

So8res
9 May 2019 19:16 UTC
69 points
22 comments5 min readLW link

Ein­stein’s Arrogance

Eliezer Yudkowsky
25 Sep 2007 1:29 UTC
67 points
88 comments3 min readLW link

A Fer­vent Defense of Fre­quen­tist Statistics

jsteinhardt
18 Feb 2014 20:08 UTC
47 points
129 comments16 min readLW link

In­finite Certainty

Eliezer Yudkowsky
9 Jan 2008 6:49 UTC
50 points
127 comments4 min readLW link

Fre­quen­tist Magic vs. Bayesian Magic

Wei_Dai
8 Apr 2010 20:34 UTC
43 points
79 comments3 min readLW link

How Much Ev­i­dence Does It Take?

Eliezer Yudkowsky
24 Sep 2007 4:06 UTC
61 points
32 comments4 min readLW link

Bayes Academy: Devel­op­ment re­port 1

Kaj_Sotala
19 Nov 2014 22:35 UTC
47 points
28 comments8 min readLW link

Multiplicitous

Jacobian
18 Dec 2016 16:39 UTC
9 points
0 comments12 min readLW link
(putanumonit.com)

Dreams with Da­m­aged Priors

Eliezer Yudkowsky
8 Aug 2009 22:31 UTC
39 points
61 comments3 min readLW link

Bayes’ The­o­rem in three pictures

Sunny from QAD
21 Jul 2019 7:01 UTC
32 points
3 comments4 min readLW link

Hearsay, Dou­ble Hearsay, and Bayesian Updates

Mass_Driver
16 Feb 2012 22:31 UTC
47 points
107 comments6 min readLW link

Prob­a­bil­ity is Sub­jec­tively Objective

Eliezer Yudkowsky
14 Jul 2008 9:16 UTC
27 points
71 comments11 min readLW link