What should you use for the “ignorance prior” for the value of a continuous random variable whose underlying distribution is unknown? (For a random variable that is both discrete and finite, you can always use the uniform distribution as your prior, but that doesn’t work when the variable can take an infinite number of values.) And you can’t always use an improper prior, either...
Would you be surprised if the absolute value was bigger than 3^^^3? I’m guessing yes, very much so. So that’s a reason not to use an improper prior.
If there’s no better information about the problem, I sortof like using crazy things like Normal(0,1)*exp(Cauchy); that way you usually get reasonable smallish numbers, but you don’t become shocked by huge or tiny numbers either. And it’s proper.
What if I know that my uncertain parameter C from a model equation C*exp(T/T0) is in a certain range… but then I wonder whether I should instead be evaluating the exact same equation as exp(T/T0 + D) (with D = ln(C), C = exp(D))? Do I use the uniform distribution on C, which will correspond to a skewed distribution on D, or do I use the uniform distribution on D which will correspond to a skewed distribution on C?
There’s a uniform prior that works just fine for this. (I think that EY’s infinite set atheism might have left you with the notion that only discrete, finite sets can have uniform priors. This is false.)
Namely, if you take any interval in [0,1] of length p, then your prior probability of X lying in that interval equals p.
Obvious corollary: your prior probability for X equaling a particular real number is zero. No amount of evidence can let you conclude the exact value of X, just narrower and narrower intervals to which it belongs. If you’re in a situation where you expect it’s possible to find out that the answer is some exact real number, then you need to take a prior which accounts for this, and averages this “continuous” prior with a countable discrete prior over the exact values that it could conceivably take. (There are several different questions and objections you could raise here, but this actually works, and I’m happy to say more if you want.)
Yeah, you can have a uniform continuous distribution on a finite interval. The problem is that it’s not actually uninformative: if you know nothing about x other than that x is in [0,1], you also know nothing about x^2 other than that it, too, is in [0,1] - but if you use the uniform distribution for x, then you’re not using a uniform distribution for x^2… I think the Jeffreys prior was supposed to solve this problem, but I don’t really understand what a Jeffreys prior is in general...
but if you use the uniform distribution for x, then you’re not using a uniform distribution for x^2
Well, x^2 isn’t an isometry, so you shouldn’t expect it to leave the prior unchanged.
Let me put it this way: if Omega told you ve had a real number x between 0 and 1, and then ve told you that x^1000 was between 3⁄10 and 4⁄10, you probably should be more surprised than if ve told you that x^1000 was between 0 and 1⁄10. Yes, you could pick your prior to have a uniform distribution for x^1000 rather than for x, but that doesn’t seem a natural choice in general.
I have also been wondering about when it’s appropriate to use a uniform prior.
As an example (I think) of an instance where a uniform prior is not appropriate: NBA stats people calculate points per minute played for all the players. In some cases, bench players have higher points per minutes played than the starters. However, it does not follow that the bench player should be starting (ignoring defensive stats).
This is because bench players tend to enter the game at a time when they will play against the opposing team’s bench. So, presuming that the defensive skills of the opponent’s bench are less than the defensive skills of the opponent’s starters, it’s clear that there is a non-uniform level of defense maintained by the opponent during the game—i.e., bench players should have an easier time scoring than starters. So there is not a simple apples to apples comparison & conclusion based on this stat.
I don’t think there is any kind of general result for this. It depends on what you’re trying to infer. For example, are you trying to find a scale parameter? Are you trying to find a shape parameter? I think the most popular approach is to find the Maximm Entropy distribution. Admittedly I don’t know a lot about the math behind this.
Math-wise, basically you pick whatever distribution maximizes integral from negative infinity to infinity of p(x)log(p(x)) with respect to x, multiplied by −1. So -Sp(x)log(p(x))dx.
The crux of this making sense is that that value can be interpreted as the amount of information you expect to learn from hearing that x happened. Or more straightforwardly, its how much you expect to not know about a particular variable/event. If you use log base 2, its measured in the average number of yes/no questions needed to concisely learn that it happened. For an explanation of why that’s true, thesearticles are excellent.
The reason that you want to maximize this value in the distribution is that not doing so assumes that you have information that you don’t know. Say you have 5 bits of entropy in the maximum entropy distribution, and 4 in some other one. If you choose the4 bit one then you’re basically making up information by thinking that you need one fewer yes/no question than you actually do.
What should you use for the “ignorance prior” for the value of a continuous random variable whose underlying distribution is unknown? (For a random variable that is both discrete and finite, you can always use the uniform distribution as your prior, but that doesn’t work when the variable can take an infinite number of values.) And you can’t always use an improper prior, either...
Would you be surprised if the absolute value was bigger than 3^^^3? I’m guessing yes, very much so. So that’s a reason not to use an improper prior.
If there’s no better information about the problem, I sortof like using crazy things like Normal(0,1)*exp(Cauchy); that way you usually get reasonable smallish numbers, but you don’t become shocked by huge or tiny numbers either. And it’s proper.
Let’s say that you know the variable is a real number in [0,1], but nothing else...
If it’s a frequency, I believe the accepted prior is (x(1-x))^(-1/2), divided by pi if you want it normalized. This is known as the Jefferys prior.
Well if it’s you know it’s in [0,1], then you’ve got a perfectly good uniform distribution.
What if I know that my uncertain parameter C from a model equation C*exp(T/T0) is in a certain range… but then I wonder whether I should instead be evaluating the exact same equation as exp(T/T0 + D) (with D = ln(C), C = exp(D))? Do I use the uniform distribution on C, which will correspond to a skewed distribution on D, or do I use the uniform distribution on D which will correspond to a skewed distribution on C?
There’s a uniform prior that works just fine for this. (I think that EY’s infinite set atheism might have left you with the notion that only discrete, finite sets can have uniform priors. This is false.)
Namely, if you take any interval in [0,1] of length p, then your prior probability of X lying in that interval equals p.
Obvious corollary: your prior probability for X equaling a particular real number is zero. No amount of evidence can let you conclude the exact value of X, just narrower and narrower intervals to which it belongs. If you’re in a situation where you expect it’s possible to find out that the answer is some exact real number, then you need to take a prior which accounts for this, and averages this “continuous” prior with a countable discrete prior over the exact values that it could conceivably take. (There are several different questions and objections you could raise here, but this actually works, and I’m happy to say more if you want.)
Yeah, you can have a uniform continuous distribution on a finite interval. The problem is that it’s not actually uninformative: if you know nothing about x other than that x is in [0,1], you also know nothing about x^2 other than that it, too, is in [0,1] - but if you use the uniform distribution for x, then you’re not using a uniform distribution for x^2… I think the Jeffreys prior was supposed to solve this problem, but I don’t really understand what a Jeffreys prior is in general...
Well, x^2 isn’t an isometry, so you shouldn’t expect it to leave the prior unchanged.
Let me put it this way: if Omega told you ve had a real number x between 0 and 1, and then ve told you that x^1000 was between 3⁄10 and 4⁄10, you probably should be more surprised than if ve told you that x^1000 was between 0 and 1⁄10. Yes, you could pick your prior to have a uniform distribution for x^1000 rather than for x, but that doesn’t seem a natural choice in general.
I have also been wondering about when it’s appropriate to use a uniform prior.
As an example (I think) of an instance where a uniform prior is not appropriate: NBA stats people calculate points per minute played for all the players. In some cases, bench players have higher points per minutes played than the starters. However, it does not follow that the bench player should be starting (ignoring defensive stats).
This is because bench players tend to enter the game at a time when they will play against the opposing team’s bench. So, presuming that the defensive skills of the opponent’s bench are less than the defensive skills of the opponent’s starters, it’s clear that there is a non-uniform level of defense maintained by the opponent during the game—i.e., bench players should have an easier time scoring than starters. So there is not a simple apples to apples comparison & conclusion based on this stat.
There’s no general answer to that question.
I don’t think there is any kind of general result for this. It depends on what you’re trying to infer. For example, are you trying to find a scale parameter? Are you trying to find a shape parameter? I think the most popular approach is to find the Maximm Entropy distribution. Admittedly I don’t know a lot about the math behind this.
Math-wise, basically you pick whatever distribution maximizes integral from negative infinity to infinity of p(x)log(p(x)) with respect to x, multiplied by −1. So -Sp(x)log(p(x))dx.
The crux of this making sense is that that value can be interpreted as the amount of information you expect to learn from hearing that x happened. Or more straightforwardly, its how much you expect to not know about a particular variable/event. If you use log base 2, its measured in the average number of yes/no questions needed to concisely learn that it happened. For an explanation of why that’s true, these articles are excellent.
The reason that you want to maximize this value in the distribution is that not doing so assumes that you have information that you don’t know. Say you have 5 bits of entropy in the maximum entropy distribution, and 4 in some other one. If you choose the4 bit one then you’re basically making up information by thinking that you need one fewer yes/no question than you actually do.