Relatedly, there’s Conservation of Expected Evidence. A rational person can’t seek to confirm their beliefs, only to test them. You should expect that, on average, a test will leave your beliefs unchanged. If not, you should update your beliefs now based on how you expect the test to turn out.
This appears to be wrong: Shake a box containing a coin. What is your belief that the coin landed heads? 50% . Will your belief change if you open the box and look inside it? Sure it will.
You should expect that, on average, a test will leave your beliefs unchanged.
Emphasis mine.
When I shake the box, my belief that the coin landed heads is 50%. When I look inside, my belief changes, yes, but two one of two options of equal probability: 0% (I see it came out tails), or 100% (I see it came out heads.)
It is trivial to see that my expected posterior belief is 0% 1⁄2 + 100% 1⁄2 = 50%, or in other words, it’s exactly equal to my prior belief.
The question is whether ‘change’ signifies only a magnitude or also a direction. The average magnitude of the change in belief when doing an experiment is larger than zero. But the average of change as vector quantity, indicating the difference between belief after and before the test, is zero.
If you drive your car to work and back, then the average velocity of your trip is 0, but the average speed is positive.
You should expect that, on average, a test will leave your beliefs unchanged.
Emphasis mine.
The statement is still wrong: Opening the box always changes your beliefs, therefore, it also changes your beliefs on average.
The correct version of this statement is “your belief over the beliefs that you will have after performing a test must be equivalent to your current belief”, which seems to be a trivial claim.
The correct version of this statement is “your belief over the beliefs that you will have after performing a test must be equivalent to your current belief”, which seems to be a trivial claim.
It may seem trivial but then again so does the claim that P(A and B) ⇐ P(A), and still...
In particular, I’ve sometimes caught myself simultaneously having aliefs like ‘if she flees, then she must be a witch’, ‘if she stays, then she must be a witch’, and ‘she may or may not be a witch, and I can’t know until I see whether she flees or stays’, and until I read the post about conservation of expected evidence I never realized there was something wrong with that.
The statement, “You should expect that, on average, a test will leave your beliefs unchanged,” means that you cannot expect an unbiased test to change you beliefs in a particular direction, as is clear from the original post.
Of course you expect to hold different beliefs after the test. If you didn’t, the test would not be worth doing. But you are not more likely to end up at (100% heads, 0% tails) than (0% heads, 100% tails).
On the other hand, if you think it is more likely that you will end up at, say, (0% heads, 100% tails), then you cannot rightly claim that you currently believe the coin to be fair (your 50%, 50% estimate does not reflect your true expectations).
That said, it’s far from the most easily accessible formulation of that meaning imaginable.
I mean, sure, the future state in which half of my measure has ~1 confidence in “heads” and half my measure has ~0 confidence in “heads” is in some sense not a change from my current state where I have .5 confidence in “heads”, but that’s not the interpretation most people will adopt of “leave your beliefs unchanged.”
It seems more accessible to say that if I expect a test to update my beliefs in a particular direction, I should go ahead and update my beliefs in that direction now (and perform the test as confirmation).
Of course, this advice presumes that I won’t anchor on my new belief. Which, given that I’m human, is not a safe assumption.
I would suggest that you expect your beliefs to be changed in 100% of cases. Currently, you believe in a 50% probability. After doing the tests, we have a set of universes, some of in which you believe a 100% probability and some of in which you believe a 0% probability. Your belief changed in every single one.
X and Y can be averaged out, but belief in number X and belief in number Y don’t average out to be “belief in the average of X and Y”.
The statement, “You should expect that, on average, a test will leave your beliefs unchanged,” means that you cannot expect an unbiased test to change you beliefs in a particular direction, as is clear from the original post.
Actually you can: Shake a box with a coin you know to be biased. Before you look into the box, your belief for heads is, say, 80%. You expect that is more likely that, when you open the box, your belief will change to 100% heads rather than 0%.
I don’t think there is an useful way to patch the statement without making explicit reference to the technical definition of Bayesian belief.
I agree that the statement is not crystal clear. It makes it possible to confuse the (change in the average) with the (average of the change).
Mathematically speaking, we represent our beliefs as a probability distribution on the possible outcomes, and change it upon seeing the result of a test (possibly for every outcome). The statement is that “if we average the possible posterior probability distributions weighted by how likely they are, we will end up with our original probability distribution.”
If that were not the case, it would imply that we were failing to make use of all of the prior information we have in our original distribution.
A misunderstood reading of the statement is that “the average of the absolute change in the probability distribution on measurement is zero.” This is not the case, as you rightly point out. It would imply that we expect the test to yield no information.
The thread descending from this comment exemplifies a pit that is easy to fall into when reading an informal moral drawn from a precise mathematical result: mistaking the former for the latter, and arguing about the former instead of going to the latter. The whole nugatory discussion would be avoided had people gone back to the original mathematics, which is not deep, and is given in one of the Sequence posts the OP linked to.
This mathematics, which is simple and straightforward, but not a complete triviality, says precisely what is meant by the informal phrase, “Conservation of Expected Evidence”, and provides an immediate answer to questions such as “but making an observation will change your belief, so you can expect your belief to change!”, or “but what about a lottery ticket, you expect that to lose, don’t you?”
There’s no point in basing an argument on secondary sources when the primary source is right there.
There’s a sense in which what I said is true (see ygert’s comment), but I agree it’s confusing. Suggested re-word? Or maybe I should just cut that point.
You should expect that, on average, a test will leave your beliefs unchanged.
That happens to be not true. A test which ouputs useful information WILL change your beliefs. Especially given point 2, one can say “Any informative test will always change your beliefs”.
What’s tricky here is expectation. You expect your beliefs to change but you don’t know in which direction. So your expectation is for zero change even though you know that you’ll get some non-zero change.
This looks paradoxical, but is the entirely standard way in which statistics (in particular random variables) operate. Consider a toss of a fair coin. The expectation is half heads half tails which is guaranteed not to happen. You know you’ll get either heads or tail but not which one of those two. The expectation will not match the outcome—all it can do is be equidistant (appropriately weighted) from all possible outcomes.
Your expectation of the possible beliefs you could have after seeing the test results should match your current belief.
Another option is to try to illustrate both CoEE and Beliefs Pay Rent in Anticipated Experiences at the same time, since I think failing BPRiAE demonstrates an easy way to fail CoEE.
All the passage says is that if you believe the coin is unbiased, then you expect to see a roughly 50-50 split between heads and tails. If you expect to see 70:30 split of heads:tails, you ought to believe that the coin is so biased before you do the experiment. It looks trivial when applied to coins, but less so in other contexts. This is a statement about priors, not posteriors, hence the term “expectation”. In Eliezer’s example, if you are p% confident that an accused is a witch, then you should expect a definitive witch test to exonerate the accused (100-p)% of the time. If any outcome “confirms witchiness”, then the test in question is not a test of witchiness.
This appears to be wrong:
Shake a box containing a coin. What is your belief that the coin landed heads? 50% . Will your belief change if you open the box and look inside it? Sure it will.
Emphasis mine.
When I shake the box, my belief that the coin landed heads is 50%. When I look inside, my belief changes, yes, but two one of two options of equal probability: 0% (I see it came out tails), or 100% (I see it came out heads.)
It is trivial to see that my expected posterior belief is 0% 1⁄2 + 100% 1⁄2 = 50%, or in other words, it’s exactly equal to my prior belief.
The question is whether ‘change’ signifies only a magnitude or also a direction. The average magnitude of the change in belief when doing an experiment is larger than zero. But the average of change as vector quantity, indicating the difference between belief after and before the test, is zero.
If you drive your car to work and back, then the average velocity of your trip is 0, but the average speed is positive.
The statement is still wrong:
Opening the box always changes your beliefs, therefore, it also changes your beliefs on average.
The correct version of this statement is “your belief over the beliefs that you will have after performing a test must be equivalent to your current belief”, which seems to be a trivial claim.
It may seem trivial but then again so does the claim that P(A and B) ⇐ P(A), and still...
In particular, I’ve sometimes caught myself simultaneously having aliefs like ‘if she flees, then she must be a witch’, ‘if she stays, then she must be a witch’, and ‘she may or may not be a witch, and I can’t know until I see whether she flees or stays’, and until I read the post about conservation of expected evidence I never realized there was something wrong with that.
The statement, “You should expect that, on average, a test will leave your beliefs unchanged,” means that you cannot expect an unbiased test to change you beliefs in a particular direction, as is clear from the original post.
Of course you expect to hold different beliefs after the test. If you didn’t, the test would not be worth doing. But you are not more likely to end up at (100% heads, 0% tails) than (0% heads, 100% tails).
On the other hand, if you think it is more likely that you will end up at, say, (0% heads, 100% tails), then you cannot rightly claim that you currently believe the coin to be fair (your 50%, 50% estimate does not reflect your true expectations).
That said, it’s far from the most easily accessible formulation of that meaning imaginable.
I mean, sure, the future state in which half of my measure has ~1 confidence in “heads” and half my measure has ~0 confidence in “heads” is in some sense not a change from my current state where I have .5 confidence in “heads”, but that’s not the interpretation most people will adopt of “leave your beliefs unchanged.”
It seems more accessible to say that if I expect a test to update my beliefs in a particular direction, I should go ahead and update my beliefs in that direction now (and perform the test as confirmation).
Of course, this advice presumes that I won’t anchor on my new belief. Which, given that I’m human, is not a safe assumption.
I would suggest that you expect your beliefs to be changed in 100% of cases. Currently, you believe in a 50% probability. After doing the tests, we have a set of universes, some of in which you believe a 100% probability and some of in which you believe a 0% probability. Your belief changed in every single one.
X and Y can be averaged out, but belief in number X and belief in number Y don’t average out to be “belief in the average of X and Y”.
Actually you can:
Shake a box with a coin you know to be biased. Before you look into the box, your belief for heads is, say, 80%. You expect that is more likely that, when you open the box, your belief will change to 100% heads rather than 0%.
I don’t think there is an useful way to patch the statement without making explicit reference to the technical definition of Bayesian belief.
I agree that the statement is not crystal clear. It makes it possible to confuse the (change in the average) with the (average of the change).
Mathematically speaking, we represent our beliefs as a probability distribution on the possible outcomes, and change it upon seeing the result of a test (possibly for every outcome). The statement is that “if we average the possible posterior probability distributions weighted by how likely they are, we will end up with our original probability distribution.”
If that were not the case, it would imply that we were failing to make use of all of the prior information we have in our original distribution.
A misunderstood reading of the statement is that “the average of the absolute change in the probability distribution on measurement is zero.” This is not the case, as you rightly point out. It would imply that we expect the test to yield no information.
The thread descending from this comment exemplifies a pit that is easy to fall into when reading an informal moral drawn from a precise mathematical result: mistaking the former for the latter, and arguing about the former instead of going to the latter. The whole nugatory discussion would be avoided had people gone back to the original mathematics, which is not deep, and is given in one of the Sequence posts the OP linked to.
This mathematics, which is simple and straightforward, but not a complete triviality, says precisely what is meant by the informal phrase, “Conservation of Expected Evidence”, and provides an immediate answer to questions such as “but making an observation will change your belief, so you can expect your belief to change!”, or “but what about a lottery ticket, you expect that to lose, don’t you?”
There’s no point in basing an argument on secondary sources when the primary source is right there.
I think the problem is that people tend to derive incorrect, or at least misleading, informal beliefs from the correct math.
http://en.wikipedia.org/wiki/Law_of_total_expectation
Expectation of your belief E(X) is not the same as your belief X.
There’s a sense in which what I said is true (see ygert’s comment), but I agree it’s confusing. Suggested re-word? Or maybe I should just cut that point.
I think that problem is in the sentence
That happens to be not true. A test which ouputs useful information WILL change your beliefs. Especially given point 2, one can say “Any informative test will always change your beliefs”.
What’s tricky here is expectation. You expect your beliefs to change but you don’t know in which direction. So your expectation is for zero change even though you know that you’ll get some non-zero change.
This looks paradoxical, but is the entirely standard way in which statistics (in particular random variables) operate. Consider a toss of a fair coin. The expectation is half heads half tails which is guaranteed not to happen. You know you’ll get either heads or tail but not which one of those two. The expectation will not match the outcome—all it can do is be equidistant (appropriately weighted) from all possible outcomes.
I might go with:
Another option is to try to illustrate both CoEE and Beliefs Pay Rent in Anticipated Experiences at the same time, since I think failing BPRiAE demonstrates an easy way to fail CoEE.
All the passage says is that if you believe the coin is unbiased, then you expect to see a roughly 50-50 split between heads and tails. If you expect to see 70:30 split of heads:tails, you ought to believe that the coin is so biased before you do the experiment. It looks trivial when applied to coins, but less so in other contexts. This is a statement about priors, not posteriors, hence the term “expectation”. In Eliezer’s example, if you are p% confident that an accused is a witch, then you should expect a definitive witch test to exonerate the accused (100-p)% of the time. If any outcome “confirms witchiness”, then the test in question is not a test of witchiness.