The idea is that the two terms in the score balance between two effects: trying to make S as small as possible means making your interval as small as possible, but if you make it too small you’re more likely to use an interval which doesn’t contain the truth. Trying to make D as small as possible means making your interval more likely to contain the truth. The coefficients balance the tradeoff between the two so that the interval you end up with is your 90% confidence interval. (According to Scott; I haven’t verified this personally.)
I have verified it. I was in the process of writing a (fairly lengthy) reply to Stefan’s comment, including a proof that Scott’s scoring rule does indeed have the property that your expected score (according to your actual beliefs about the quantity you’re estimating) is maximized when the confidence interval you state has (again according to your actual beliefs) a 5% chance that the quantity lies below its lower bound and a 5% chance that the quantity lies above its upper bound … but then something I did (I have no inkling what, though it coincided with some combination of keypresses as I was trying to enter some mathematics) made the page go entirely blank, and I didn’t find any way to get my partially-written comment back again.
Anyway, here’s one way (I don’t guarantee it’s best and it feels like there should be a slicker way) to prove it. Let’s suppose the confidence interval you state is (l,r); consider the derivative w.r.t. either of those bounds—let’s say r, but l is similar—of your expected score. The first term in the score is just l-r, and the derivative of that is always −1. The second term can be written as an integral; differentiating it w.r.t. r turns out to give you 20Pr(X>r). (The calculation is easy.) So the derivative is zero only when 1-20Pr(X>r)=0; that is, when Pr(X>r)=5%. So if the confidence interval you state doesn’t have the property that you expect to be above it exactly 5% of the time, then this derivative is nonzero and therefore some small change in r increases your expected score.
Suppose f is your probability density function for the quantity X you’re interested in.
Then the expectation of D is the integral of D(x)f(x), which equals the integral of [max(0,l-x)+max(0,x-r)]f(x). When we differentiate w.r.t. r, the first term obviously goes away because it’s independent of r, so we get the integral of [d/dr max(0,x-r)] f(x). That derivative is 0 for x<r and 1 for x>r, so this is the integral of f(x) from r upwards; in other words it’s Pr(X>r). So d(score)/dr = 1-20Pr(X>r).
The calculation for l is exactly the same but with a change of sign; we end up with 20Pr(X<l)-1.
The idea is that the two terms in the score balance between two effects: trying to make S as small as possible means making your interval as small as possible, but if you make it too small you’re more likely to use an interval which doesn’t contain the truth. Trying to make D as small as possible means making your interval more likely to contain the truth. The coefficients balance the tradeoff between the two so that the interval you end up with is your 90% confidence interval. (According to Scott; I haven’t verified this personally.)
I have verified it. I was in the process of writing a (fairly lengthy) reply to Stefan’s comment, including a proof that Scott’s scoring rule does indeed have the property that your expected score (according to your actual beliefs about the quantity you’re estimating) is maximized when the confidence interval you state has (again according to your actual beliefs) a 5% chance that the quantity lies below its lower bound and a 5% chance that the quantity lies above its upper bound … but then something I did (I have no inkling what, though it coincided with some combination of keypresses as I was trying to enter some mathematics) made the page go entirely blank, and I didn’t find any way to get my partially-written comment back again.
Anyway, here’s one way (I don’t guarantee it’s best and it feels like there should be a slicker way) to prove it. Let’s suppose the confidence interval you state is (l,r); consider the derivative w.r.t. either of those bounds—let’s say r, but l is similar—of your expected score. The first term in the score is just l-r, and the derivative of that is always −1. The second term can be written as an integral; differentiating it w.r.t. r turns out to give you 20Pr(X>r). (The calculation is easy.) So the derivative is zero only when 1-20Pr(X>r)=0; that is, when Pr(X>r)=5%. So if the confidence interval you state doesn’t have the property that you expect to be above it exactly 5% of the time, then this derivative is nonzero and therefore some small change in r increases your expected score.
would you mind spelling out the integral part?
Suppose f is your probability density function for the quantity X you’re interested in.
Then the expectation of D is the integral of D(x)f(x), which equals the integral of [max(0,l-x)+max(0,x-r)]f(x). When we differentiate w.r.t. r, the first term obviously goes away because it’s independent of r, so we get the integral of [d/dr max(0,x-r)] f(x). That derivative is 0 for x<r and 1 for x>r, so this is the integral of f(x) from r upwards; in other words it’s Pr(X>r). So d(score)/dr = 1-20Pr(X>r).
The calculation for l is exactly the same but with a change of sign; we end up with 20Pr(X<l)-1.
Thanks for this reply. The technique of asking what each term of your equation represents is one I have not practiced in some time.
This answer very much helped me to understand the model.