The Variational Characterization of Expectation

Epistemological Status: Fairly confident. This is much closer to the expected minimal map I had in mind ;)

Expectation for a random variable $X$ given $Y$ is the best point estimate of $X$ given $Y$ under squared error loss.

Say we have a random variable $X$ , but are only allowed to summarize the outcomes with a single number $e_{X}$ . What should we pick so that the squared error is minimized? $min e_{X} E [(X - e_{X})^{2}] ⟺ \partial_{e_{X}} E [(X - e_{X})^{2}] = 0 ⟺ E [X - e_{X}] = 0 ⟺ E [X] = e_{X}$ Thus, in this sense, the expectation of a random variable is the best point estimate of it’s outcomes.

A major reason a variational characterization is interesting is because it creates a tunnel to optimization theory. The expected minimal map presented here can ‘average out’ irrelevant details allowing you to filter to just the relevant things in a stream-lined manner.

When the random variables have binary outcomes we can use the conditional expectation to characterize probability without referencing information. It is effectively the same statement without reference to information. So there are at least two ways of giving a variational characterization of probability.

Lemma 1 (Optimal Prediction): Define a random variable $χ_{U} \in L_{2}$ . The best point representation of the outcome of $χ_{U}$ for a given observation $O$ is equivalent to $E (χ_{U} | O)$ . Moreover, the optimal point representation of $χ_{U}$ is invariant under the pull-back $η : O \to E (χ_{U} | O)$ of the conditioning.

Corollary: When $χ_{U}$ indicates a binary answer to a query we have a characterization of probability.

The second condition is a fancy way of saying that our optimal estimate is an optimal way to condition our expectation. It’s somewhat like cheating on a test. You can either read the question $Q$ and then predict $A$ via $E [A | Q]$ or you could be told the answer and then predict $A$ via $E [A | E [A | Q]]$ .

Suppose that instead of cheating we answer some other question that gives us a hint for the answer. Now we have something like, $Q \to A_{1} \to A_{2}$ We have a question, we get a hint, we answer. From the above lemma, the best guess for $A_{2}$ is $E [A_{2} | A_{1}]$ , but we don’t know $A_{1}$ . Our best guess for $A_{1}$ is $E [A_{1} | Q]$ so the best we can do is, $E [A_{2} | Q] \approx E [A_{2} | E [A_{1} | Q]]$ On a multi-part problem we might having something like, $(Q = A_{0}) \to A_{1} \to \dots \to A_{n - 1} \to A_{n}$ So we take the approximations as, ${^A}_{k} = E [A_{k} | {^A}_{k - 1}] = {argmin}_{e_{k} \in L_{2}} E [(A_{k} - e_{k} ({^A}_{k - 1}))^{2}]$ In practice, we’ll often want or need to restrict the class of functions we know how to optimize over. In such cases extending the Q/A process is the only way to guarantee that the final answer is always close to the optimum. At this point we can ‘forget’ about the intermediates and optimize, ${^A}_{n} = {argmin}_{e_{1}, \dots, e_{n} \in E} E [(A_{n} - e_{n} \circ \dots \circ e_{2} \circ e_{1} (Q))^{2}]$ If the function class is appropriate, this gives you the optimization problem associated with training a neural-network. Literally, but only approximately, each function averages out irrelevant features of it’s input to create more relevant features for prediction in it’s output.

Proof of Lemma 1: The proof works for non-binary random variables. However, the probabalistic interpretation is lost. Expectation for random variables in $L_{2}$ is characterized by, $P (χ_{U} | O) = E [χ_{U} | O] = min e_{χ_{U}} \in L_{2} E [(χ_{U} - e_{χ_{U}} (O))^{2}]$ The minimizer exists and is unique by the Hilbert projection theorem. Moreover, $E [χ_{U} | η (O)] = E [χ_{U} | E (χ_{U} | O)] ⟺ {argmin}_{e_{χ_{U}} \in L_{2}} E [(χ_{U} - e_{χ_{U}} (O))^{2}] = {argmin}_{e_{1} \in L_{2}} E [(χ_{U} - e_{1} (E [χ_{U} | O]))^{2}]$ So all that’s left is to verify the pull-back doesn’t affect the outcome. We have, ${argmin}_{e_{1} \in L_{2}} E [(χ_{U} - e_{1} (E [χ_{U} | O]))^{2}] = {argmin}_{e_{1} \in L_{2}} E [{(χ_{U} - e_{1} ({argmin}_{e_{2} \in L_{2}} E [(χ_{U} - e_{2} (O))^{2}]))}_{U}^{2}] = {argmin}_{e_{1} \in L_{2}} E [{(χ_{U} - e_{1} \circ e_{2} (O))}_{U}^{2}] = {argmin}_{e_{1}, e_{2} \in L_{2}} E [{(χ_{U} - e_{1} \circ e_{2} (O))}_{U}^{2}] = {argmin}_{e_{χ_{U}} \in L_{2}} E [{(χ_{U} - e_{χ_{U}} (O))}_{U}^{2}]$ Therefore, the probability conditioned on the observation is the optimal point estimate and pulling-back the observation to just the point-estimate leaves the estimate unchanged. $□$