Insights from “All of Statistics”: Probability

[Rafael Harth]

All of Statistics is yet another book on Miri’s research guide. This book has also been reviewed by Alex Turner.

This post is something like an abridged summary. It’s definitely not a self-contained introduction to the field; I’ve included all important subjects as sections mostly to remind myself, but I skip over many of them and make no effort to introduce every concept I write about. The primary purpose is learning by writing; beyond that, it may be useful for people already familiar with statistics to refresh their knowledge.

I’m also including more of my own takes than I’ve done in my previous posts. For example, if I sound dismissive of Frequentism, this is not the book’s fault.

My verdict on the book’s quality that it’s one of the weakest ones Miri’s guide (but the competition is tough). This is mostly based on my recurring feeling that ‘this could have been explained better’. Nonetheless, I’m not familiar with a better book on statistics.

The book is structured in three parts: (1) Probability, (2) Statistical Inference, and (3) Statistical Models and Methods. Since this post was getting too long, I’ve decided to structure my summary in the same way: this post covers probability, the other post covers Statistical Inference, and there may be a third post in the future covering Statistical Models and Methods.

1. Probability spaces and distributions

The general notion of a probability space includes $\sigma$-algebras, but the book gets by without them. Thus, a probability space is either discrete with a probability mass function (pmf) or continuous with a probability density function (pdf). I denote both by writing $f$. In both cases, $f$ determines a distribution $P$ that assigns probability mass to subsets of the sample space $\Omega$.

1.1. Events

1.2. Independence

1.3. Finite Sample Spaces

1.4. Conditional Probability

Given two events $A,B\subset \Omega$, the formula everyone learns in high school is

$$P\left(A|B\right)=\frac{P(A\cap B)}{P\left(B\right)}$$

In university courses, it may also be mentioned that $P(\cdot |B)$ is a probability space for any nonempty set $B\subset \Omega$.

I prefer to take that as the starting point, i.e., “evaluate event $A$ in the probability distribution over $B$” rather than “plug $A$ into the formula above”. This seems to be what people do when they’re not trying to do math. If I hear that a die came up either $2$ or $3$ or $5$ and consider the probability that it’s at least a $5$, I don’t think “let’s take the intersection of $\{5,6\}$ and $\{2,3,5\}$; let’s compute $P\left(\right\{5\left\}\right)$; let’s compute $P\left(\right\{2,3,5\}$; let’s divide $\frac{1/6}{1/2}$ to get $\frac{1}{3}$”. Instead, I think something like, “it can’t be 6, there are three remaining (equally probable) possibilities, so it’s $\frac{1}{3}$”. Because of this, I like the notation $P_B\left(A\right)$ rather than $P\left(A|B\right)$.

1.5. Joint Distributions

1.6. Marginal Distributions

1.7. Bayes’ Theorem

See this amazing guide.

2. Random Variables

2.1. Fundamentals

A random variable (RV) is a way to measure properties of events that we care about. If $\Omega =\{1,2,3,4,5,6{\}}^5$ denotes all results of throwing five dice, a random variable may count the sum of results, the maximum result, the minimum result, or anything else we are interested in.

The traditional formalism defines a RV as a function $X:\Omega \to R$.

In advanced probability theory, the sample spaces soon slide into the background. To see why, consider a random variable $X$ that measures how often patients get surgery. In this case, the probability space $\Omega$ of $X$ is the set of all possible data sets that we could have obtained by collecting data. Trying to define and deal with this space is impractical, so instead, one generally talks in terms of RVs and their distributions.

To define this formally, let $X:\Omega \to R$ with probability measure $P$ on $\Omega$. Then, we get a new probability space ${\Omega }^{\prime }:=X\left(\Omega \right)=R$ with probability measure $P^{\prime }$ defined by $P^{\prime }\left(A\right):=P\left({\Omega }^{-1}\right(A\left)\right)$ for any $A\subseteq R$. This works regardless of whether $P$ is given by a pmf or pdf. In fact, by the time one gets to continuous spaces, the sample space has already disappeared.

There are two related points I’ve been thinking about while working through the book. The first is a formal detail that most people wouldn’t care about (perhaps because I am an extreme type 1 mathematician). Given the formal definition of a RV as a function $X:\Omega \to R$, the RV doesn’t ‘know’ its distribution. It also doesn’t know enough to determine the value of statistical functionals like $E\left(X\right)$. Now consider the notation $X\sim \text{Bernoulli}\left(p\right)$. This looks like a statement about $X$, but it doesn’t determine what function $X$ is—in fact, the only constraint it puts on the sample space $\Omega$ is that $|\Omega |>1$. In this formalism, $X\sim \text{Bernoulli}\left(p\right)$ is really a statement about the distribution of $X$.

To resolve this, I’ve transitioned to thinking of any random variable $X$ as a pair $(P,X^{\prime })$, where $X^{\prime }:\Omega \to R$ is the usual function and $P$ the entire (discrete or continuous or general) probability space. Everyone already talks about RVs as if they knew their distributions, so this formalism simply adopts that convention. Now, $X\sim \text{Bernoulli}\left(p\right)$ really is a statement about $X$. The world is well again.

The second point is that the distribution does not fully characterize a RV, and this begins to matter as soon as random variables are combined. To demonstrate this, suppose we have two RVs $X,Y$. Let $f_X$ by the pdf of $X$ and $f_Y$ be the pdf of $Y$. What is the pdf of $X+Y$ (the RV defined by $\omega \mapsto X\left(\omega \right)+Y\left(\omega \right)$)? This question is impossible to answer without knowing how $X$ and $Y$ map elements from their probability space onto $R$: For example, suppose $X,Y\sim N(0,1)$:

• if $Y=X$, then $X+Y$ is continuous with pdf $f_{X+Y}\left(x\right)=\frac{1}{\sqrt{8\pi }}{e}^{-\frac{1}{8}{x}^2}$

• if $Y=-X$ (this does not violate the assumption that $X$ and $Y$ are equally distributed), then $X+Y$ is discrete with pmf $f_{X+Y}\left(0\right)=1$ and $f_{X+Y}\left(x\right)=0$ for $x\ne 0$

• if $X$ and $Y$ are independent, then $X+Y$ is continuous with pdf $f_{X+Y}\left(x\right)=\frac{1}{\sqrt{4\pi }}{e}^{-\frac{1}{4}{x}^2}$

In practice, one either has the assumption that $X$ and $Y$ are independent (which is defined by the condition that, for any sets $A,B\subset R$, the events $X^{-1}\left(A\right)$ and $Y^{-1}\left(B\right)$ are independent), or one constructs $Y$ as a function of $X$ as above. However, those are only a small subset of all possible relationships that two RVs can have. In general, the distribution of $X+Y$ can take arbitrary forms. Thus, one needs to know both the distribution and the mapping to characterize a RV.

2.2. Important Distributions

I think it’s an underappreciated aspect of probability theory how much it builds on top of itself. So many of the important distributions are derived out of prior distributions.

2.2.1. Bernoulli

The simplest possible distribution is given by $Pr(X=1)=p$ and $Pr(X=0)=1-p$. We write $X\sim \text{Bernoulli}\left(p\right)$. As with all of these, we don’t specify the sample space $\Omega$.

2.2.2. Binomial

A Binomial distribution is a Bernoulli distribution sampled $n$ times where $X$ counts the number of samples that came up 1. I call them “hits” and the others “misses”. We write $X\sim \text{Binomial}(n,p)$ and have the formula

$$P(X=k)=(\frac{n}{k})p^k(1-p{)}^{n-k}$$

It’s not too difficult to reason why this formula has this exact form.

2.2.3. Geometric

Instead of repeating Bernoulli experiments a fixed number of times, we now repeat them until we miss once. We write $X\sim \text{Geom}\left(p\right)$ and have the formula

$$P(X=k)=p^k(1-p)$$

Unfortunately, people couldn’t agree on whether we should count the number of hits or the number of samples (which is always one more), so there are two forms with slightly different formulas. In the above, I count the number of hits.

2.2.4. Poisson

Instead of repeating Bernoulli experiments a fixed number of times, or until we get one miss, we now repeat them infinitely often but decrease the probability of hitting so that the expected number of hits remains constant. Our hit probability $p$ becomes $\frac{\lambda }{n}$ where $\lambda$ is the expected number of hits. We write $X\sim \text{Poisson}\left(\lambda \right)$ and have the formula

$$P(X=k)=\underset{n\to \infty }{lim}(\frac{n}{k})\left(\frac{\lambda }{n}{)}^k\right(1-\frac{\lambda }{n}{)}^{n-k}=e^{-\lambda }\frac{{\lambda }^k}{k!}$$

Computing this limit is not trivial but doable.

These are all discrete distributions. The first two are finite, and the latter two countably infinite. The step to continuous distributions is a bigger change.

2.2.5. Normal

Instead of repeating Bernoulli experiments a fixed number of times, or until we get a miss, or infinitely often while holding the expected number of hits constant, we now repeat it infinitely often without holding the expected number of hits constant. If $p$ remains the parameter of our Bernoulli distribution, we get

$$P(X=x)=\underset{n\to \infty }{lim}(\frac{n}{k})p^k(1-p{)}^{n-k}=\frac{1}{\sqrt{\pi np(1-p)}}{e}^{-\frac{1}{2}\frac{(k-np{)}^2}{2np(1-p)}}$$

for $k$ close to $np$.^[1]

The mean and variance of a binomial distribution are $np$ and $np(1-p)$, respectively. That means one parameter is enough to control them both, and we can see both of those terms in the formula above. However, it turns out you can also tweak this distribution so that they come apart.

For a proper normal distribution, we write $X\sim N(\mu ,{\sigma }^2)$ and have the equation

$$f\left(x\right)=\frac{1}{\sqrt{2\pi \sigma }}{e}^{-\frac{1}{2}(\frac{x-\mu }{\sigma }{)}^2}$$

and this is, by many measures, the most important distribution. Unfortunately, it’s also one of the hardest to compute probabilities with analytically.

2.2.6. Uniform

With constant pdf $f\left(x\right)=\frac{1}{b-a}$ in $[a,b]$, and $f\left(x\right)\equiv 0$ for $x\notin [a,b]$. We write $X\sim \text{Uniform}(a,b)$

2.2.7. Beta

With pdf $f\left(x\right)=\frac{\Gamma (\alpha +\beta )}{\Gamma \left(\alpha \right)\Gamma \left(\beta \right)}{x}^{\alpha -1}(1-x{)}^{\beta -1}$ where $\Gamma$ is the Gamma function. I’m yet to need the explicit formula for $\Gamma$.

Among other things, the beta distribution can describe

• the distribution of the $k$-th highest number across $n$ RVs sampled with the uniform distribution

• if $X\sim \text{Bernoulli}\left(p\right)$ with $p$ unknown, the pdf of $p$ after doing a Bayesian update on samples of $X$, provided that the prior of $p$ was either uniform or itself a Beta distribution.

2.2.8. Exponential

2.2.9. Gamma

2.2.10. Cauchy

2.3. Statistical Functionals

A statistical functional is an operator from the distribution of one or more random variable(s) to $R$. In this case, we consider only statistical functionals that return a single real number that provides some information about the distribution. Statistical functionals aren’t functions because you can’t define their domain, but I believe they are what’s called class functions in set theory.

You might think that only the expected value and variance are important, but in fact, there are a surprising number of relevant statistical functionals. For most of them, closed formulas expressing their results in terms of the distribution’s parameters are known (and in many cases, deriving them isn’t too difficult).

2.3.1. Expectation

A possible definition for the expected value is

$$E\left(X\right):=\sum _{\omega \in \Omega }P\left(\omega \right)\cdot X\left(\omega \right)$$

(This also has a continuous version). Of course, as I’ve gone over before, no one thinks in terms of ‘events from the sample space and their probability’, but only ‘possible outcomes of $X$ and their probability’. Thus, we instead have the formula

$$E\left(X\right)=\{\begin{array}{cc}{\int }_{X\left(\Omega \right)}xf\left(x\right)\text{d}x& ;X\text{is continuous}\\ {\sum }_{x\in X\left(\Omega \right)}xf\left(x\right)& ;X\text{is discrete}\end{array}$$

One can write $R$ instead of $X\left(\Omega \right)$, but I like this version as a reminder that we’re looking at image points.

An advantage of this formula is the following result:

$$E\left(g\right(X\left)\right)=\{\begin{array}{cc}{\int }_{X\left(\Omega \right)}g\left(x\right)f\left(x\right)\text{d}x& ;X\text{is continuous}\\ {\sum }_{x\in X\left(\Omega \right)}g\left(x\right)f\left(x\right)& ;X\text{is discrete}\end{array}$$

Again, this works because we already think in terms of the function that gives the probability for results. For the original expectation, we ask “for each outcome $x$, multiply $x$ with the probability of $x$”; now, we ask “for each outcome $x$, multiply the transformed outcome $g\left(x\right)$ with the probability of $x$”.

One often writes ${\mu }_X$ or just $\mu$ for $E\left(X\right)$

The equation $E\left(XY\right)=E\left(X\right)E\left(Y\right)$ holds whenever $X$ and $Y$ are independent. On the other hand, the equation $E(X+Y)=E\left(X\right)+E\left(Y\right)$ always holds, and I find that people (in everyday life) tend to make arguments that directly contradict this rule. One example is the argument that we should be indifferent between two actions because most of the relevant variables are unknown. (The known ones already increase the expected value of one action over the other, unless you specifically suspect the unknown ones to skew into the other direction.) A related argument is that high error bars/​a possibility of failure implies that taking an action isn’t worth it.

2.3.2. Variance

One thing I would do if I were teaching a class on probability is to ask students how the ‘variance’ of a RV should be defined before revealing the textbook definition. One might reasonably guess $E(|X-E(X\left)|\right)=E(|X-\mu |)$, but mathematicians have an immortal preference of squaring over taking absolute values, so it’s $E\left(\right(X-\mu {)}^2)$ instead.^[2] One can prove that $V\left(X\right)=E\left(X^2\right)-E(X{)}^2$.

2.3.3. Standard Deviation

This is just the square of the variance. One often writes ${\sigma }_X$ or just $\sigma$ for $\sqrt{V\left(X\right)}$.

2.3.4 Median

The median is the number at the center of the distribution, i.e., the $x$ such that $F\left(x\right)=\frac{1}{2}$. Since such an $x$ may not exist if $F$ is discrete p, we define it as ${inf}_{x\in R}\left\{F\right(x\left)|F\right(x)>\frac{1}{2}\}$. This means we round up, so that the “median” for $X\sim \text{Binom}\left(p\right)$ is 1.

2.3.5 Mode

The mode is the most likely single point of the distribution. It’s easy to see that an arbitrarily small change to the distribution can have an arbitrarily large impact on the mode, making it much less informative than the mean or median. It’s defined as ${\text{argmax}}_{x\in R}f\left(x\right)$ where $x$ is the pdf or pmf. There are distributions for which no closed formula for the mode is known.

2.3.6. Covariance

One often sees the term $X-\mu$, which is the RV obtained by shifting $X$ so that it now centers around $0$ and is hence called “centered”. The Covariance of two RVs $X$ and $Y$ is defined as the expected product of two points from their centered distributions, i.e.,

$$\text{Cov}(X,Y)=E\left[\right(X-{\mu }_X\left)\right(Y-{\mu }_Y\left)\right]$$

2.3.7. Correlation

The Covariance tells us whether two data sets are positively or negatively correlated. However, it does not tell us how much they’re correlated because the Covariance increases with the variance of either variable. (For example, $\text{Cov}(X,2Y)=2\text{Cov(X,Y)}$, but the thing we mean by ‘correlation’ should stay the same if we rescale one of the RVs.) We can obtain the correlation by normalizing the Covariance, i.e.,

$$\text{Corr}(X,Y)=\frac{\text{Cov}(X,Y)}{{\sigma }_X{\sigma }_Y}$$

Thus, the Covariance is like the inner product $⟨x,y⟩$ on a vector space, and the Correlation like $\frac{⟨x,y⟩}{||x||||y||}$. The correlation ranges between $-1$ and $1$.

2.2.8. Skewness

The skewness measures the asymmetry of a distribution. Symmetrical distributions such as $N(0,{\sigma }^2)$ or the uniform distribution have zero skewness.

$$\kappa \left(X\right)=\int (\frac{x-\mu }{\sigma }{)}^{3}f\left(x\right)\text{dx}$$

2.3.9. Entropy

As I understand it, the entropy measures the expected amount of information obtained by sampling the distribution once. The entropy is zero if and only if the RV is a constant. For a Bernoulli variable, the entropy is 1 and also maximal for $p=0.5$.

In the discrete case with pmf $f$, we have

$$H\left(X\right)=-\sum _{y\in X\left(\Omega \right)}f\left(y\right)\log f\left(y\right)$$

Where the base of the logarithm determines whether we measure the information in bits or something else. As of now, I don’t fully understand how/​why this definition measures information. Non-discrete numbers of bits are weird.

2.3.10. Moments

The $k$-th moment of a RV $X$ is defined as

$${\int }_{-\infty }^{\infty }{x}^{k}f\left(x\right)\text{d}x$$

The $k$-th central moment is defined as

$${\int }_{-\infty }^{\infty }[x-\mu {]}^{k}f(x)\text{d}x$$

The $k$-th standardized moment is defined as

$${\int }_{-\infty }^{\infty }(\frac{x-\mu }{\sigma }{)}^{k}f\left(x\right)\text{d}x$$

The expectation is the first moment, the variance is the second central moment, and the skewness is the third standardized moment. I still don’t quite know how to think about moments.

2.4. Independence of RVs

2.5. Multivariate Distributions

2.5.1. Marginal Distribution

2.6. Conditional Distributions

2.7. Transformations of RVs

2.8. Conditional Expectation and Variance

Given RVs $X$ and $Y$, the function $g$ defined by $g\left(y\right):=E\left(X_{[Y=y]}\right)$ is a RV since

$$g:\Omega \to Rg:\omega \mapsto E\left(X_{[Y=\omega ]}\right)$$

Thus, it makes sense to compute $E\left(g\right)$ or $V\left(g\right)$. Most people don’t insist on writing it this way, so one finds just $E\left(X|Y\right)$ instead (yuck!). We have the formula

$$E\left(g\right)"="E\left(E\right(X|Y\left)\right)=E\left(X\right)$$

which feels like an extension of the law of total probability. Furthermore, we can define $h$ by $h\left(y\right):=V\left(X_{[Y=y]}\right)$. In this case, we have the formula

$$V\left(X\right)=V\left(g\right)+E\left(h\right)"="V\left(E\right(X|Y\left)\right)+E\left(V\right(X|Y\left)\right)$$

I’m assuming this would make sense if one were to spend enough time thinking about it.

2.9. Inequalities

2.9.1. Markov

The probability that any RV ends up being at least $k$ times its mean can at most be $\frac{1}{k}$. (If it were more likely than that, this alone would imply that the mean is greater than itself). In symbols, $P(X\ge \frac{k}{\mu })\le \frac{1}{k}$. The other inequalities are much more complicated.

2.9.2. Tshebycheff

2.9.3. Hoeffding

2.9.4. Jensen

As mentioned before, this says that $E\left(g\right(X\left)\right)\le g\left(E\right(X\left)\right)$ for $g$ convex—also, the opposite is true for $g$ concave. The only functions both convex and concave are affine linear functions, so the linearity of the expectation, $E(aX+c)=aE\left(X\right)+c$, can be considered a special case of Jensen’s inequality.

2.9.5. Cauchy-Schwartz

It turns out my vector space analogy is more than an analogy. The Cauchy-Schwartz inequality says that, if $⟨⟩$ is an inner product on a vector space, then

$$|⟨x,y⟩|\le ||x||||y||$$

Take RVs as vectors, define $⟨X,Y⟩:=E\left(XY\right)$ (easy to verify that this is an inner product), and we obtain

$$|E\left(XY\right)|\le \sqrt{E\left(X^2\right)E\left(Y^2\right)}$$

which is identical to $|\text{Cov}(X,Y)|\le {\sigma }_X{\sigma }_Y$ if $X$ and $Y$ are centered. The book presents this inequality with $E\left(|XY|\right)$ (which is strictly larger than $|E\left(XY\right)|$ because $|x|$ is convex) on the left side. I don’t know how one shows that this stronger inequality also holds.

3. Convergence of Random Variables

3.1. Fundamentals

The limiting behavior of sequences of RVs turns out to be extremely important in statistics because it tells us what we can know in the limit of infinite data. We have three definitions of decreasing strength (meaning that each implies the next) for convergence. All three translate what it means for a sequence of RVs to converge to a statement about what it means for numbers to converge.

• $(X_1,...,X_n)\stackrel{\text{qm}}{\to }X$ iff $E\left(\right(X-X_n{)}^2)\stackrel{n\to \infty }{⟶}0$. (Convergence in quadratic mean.)

• $(X_1,...,X_n)\stackrel{p}{\to }X$ iff $\forall ϵ>0:Pr(|X-X_n|>ϵ)\stackrel{n\to \infty }{⟶}0$. (Convergence in probability.)

• $(X_1,...,X_n)⇝X$ iff $F_n\left(x\right)\stackrel{n\to \infty }{⟶}F\left(x\right)$ at all points $x$ at which $F$ is continuous (where $F$ is the cdf of $X$). (Convergence in distribution.)

Convergence in distribution doesn’t imply convergence in probability: here is an example where it matters that a RV isn’t determined by its distribution. For $X$ symmetrical, $(-X,...,-X)$ “converges” in distribution to $X$, but it sure doesn’t converge in probability.

Convergence in probability doesn’t imply convergence in quadratic mean: you can construct a RV with increasingly slim chances for increasingly large profits. Let $X_n$ be the RV for the game “flip $n$ coins, win $2^n$ dollars if all come up heads” and we get a counter-example (where each $X_i$ has expected value 1, but $(X_1,...,X_n)$ converges in probability to the RV that is constant 0).

3.2. Theorems

The two important theorems are called the Law of Large Numbers and The Central Limit theorem. Let $(X_1,...,X_n)$ be a sequence of i.i.d. RVs. Both theorems are making a statement about the “sample mean”, ${\overline{X}}_n:=\frac{1}{n}{\sum }_{k=1}^n{X}_i$ of the distribution.

3.2.1. The Weak Law of Large Numbers

This theorem says that $(X_1,...,X_n)\stackrel{p}{\to }{C}_{\mu }$, where $C_{\mu }\equiv \mu$ is the RV that’s always $\mu$, and $\mu =E\left(X_i\right)$ (since the $X_i$ are identically distributed, they all have the same mean). I.e., if one averages across more and more samples, the variance goes to zero.

3.2.2. The Central Limit theorem

I used to be confused about this theorem, thinking that it said the following:

$${\overline{X}}_n⇝Z\text{where}Z\sim N(\mu ,\frac{{\sigma }^2}{n})$$

This equation (which is not the central limit theorem) appears to say that ${\overline{X}}_n$ converges in distribution toward a normal distribution that has increasingly small variance. However, the normal distribution with increasingly small variance just converges toward the constant distribution—also, we already know from the Law of Large Numbers that ${\overline{X}}_n$ converges in probability, and hence also in distribution, toward $C_{\mu }$.

… in fact, the law as stated is not just wrong but ill-defined because $Z$ is not a constant distribution, so ${\overline{X}}_n$ “converges” toward a moving target.

The proper Central Limit theorem is as follows: if $V\left(X_i\right)={\sigma }^2\in R$, then

$$\sqrt{n}({\overline{X}}_n-\mu )⇝Z\text{where}Z\sim N(0,{\sigma }^2)$$

That is, the distribution of ${\overline{X}}_n$ does converge toward a constant, but if we scale it up by $\sqrt{n}$ (and hence scale up its variance by $n$), then it converges toward a normal distribution (with non-moving variance).

In practice, the useful part of the Central Limit Theorem is that we can approximate $\sqrt{n}{\overline{X}}_n$ by $Z$ even if $n$ is a constant. (Otherwise, it’s unclear why we should care about $\sqrt{n}{\overline{X}}_n$, and we already know the limiting behavior of ${\overline{X}}_n$.) This technically doesn’t follow from the theorem as-is since it makes no statements about how fast $\sqrt{n}{\overline{X}}_n$ converges, but there is also a quantitative version. And for a constant $n$, the formulations $\sqrt{n}({\overline{X}}_n-\mu )\approx N(0,{\sigma }^2)$ and ${\overline{X}}_n\approx N(\mu ,\frac{{\sigma }^2}{n})$ and $\sqrt{n}{\overline{X}}_n\approx N(\sqrt{n}\mu ,{\sigma }^2)$ are indeed equivalent.

While converge in distribution doesn’t imply convergence in probability in general, it does do so if the limiting RV is a constant. Thus, the Weak Law of Large Numbers would immediately follow from the Central Limit theorem, except that the Weak Law of Large Numbers also applies in cases where the $X_i$ have infinite variance. An example of a RV with finite mean but infinite variance is the following game: “flip a coin until it gets tails; receive $\sqrt{2^n}$ dollars where $n$ is the number of heads”.

A consequence of the ‘practical’ version of the central limit theorem is that the binomial distribution can be approximated by a normal distribution. Let $X\sim \text{Binom}(n,p)$. Then $X={\sum }_{k=1}^n{X}_i$ with $X_i\sim \text{Bernoulli}\left(p\right)$. Right now, $X$ just sums up these variables without dividing by $n$, but it’s trivial verify that $X=n\cdot {\overline{X}}_n=\sqrt{n}\left(\sqrt{n}{\overline{X}}_n\right)$. Thus, since $\sqrt{n}{\overline{X}}_n\approx N(\sqrt{n}p,p(1-p\left)\right)$ (here I’ve used that $\mu =p$ and ${\sigma }^2=p(1-p)$ for Bernoulli Variables), we have $X\approx N(np,np(1-p\left)\right)$.

Here’s an example. Note that, if $Z\sim N(0,1)$ (it’s common practice to write everything in terms of a standard normal RV and call that Z), then $\frac{X-np}{\sqrt{np(1-p)}}\approx Z$. Thus, if we want to know the probability that at most 4900 out of 10000 coins come up heads, we can compute

$$Pr(X\le 4900)=Pr(X-5000\le -100)=Pr(\frac{X-5000}{50}\le -2)\approx Pr(Z\le -2)$$

which gives $\approx 0.0275$ according to WolframAlpha.

3.3 Convergence under Transformation

In which we study what can be said about $g\left(\overline{X}\right)$, where $g$ is a function.

---

1. ↩︎

   I don’t fully understand how this can be made rigorous, but it’s a known theorem so the idea is valid.

2. ↩︎

   The former definition is known as the average absolute deviation, but regular variance is far more dominant.