mattmacdermott comments on Nathan Young’s Shortform

[mattmacdermott]

I might have misunderstood you, but I wonder if you’re mixing up calculating the self-information or surpisal of an outcome with the information gain on updating your beliefs from one distribution to another.

An outcome which has probability 50% contains $-\log \left(0.5\right)=1$ bit of self-information, and an outcome which has probability 75% contains $-\log \left(0.75\right)=0.41$ bits, which seems to be what you’ve calculated.

But since you’re talking about the bits of information between two probabilities I think the situation you have in mind is that I’ve started with 50% credence in some proposition A, and ended up with 25% (or 75%). To calculate the information gained here, we need to find the entropy of our initial belief distribution, and subtract the entropy of our final beliefs. The entropy of our beliefs about A is $-p\left(A\right)\log \left(p\right(A\left)\right)-p\left(\neg A\right)\log \left(p\right(\neg A\left)\right)$.

So for 50% → 25% it’s $$H\left(P_i\right)-H\left(P_f\right)=(-0.5\log (0.5)-0.5\log (0.5\left)\right)-(-0.25\log (0.25)-0.75\log (0.75\left)\right).$$

And for 50%->75% it’s $$H\left(P_i\right)-H\left(P_f\right)=(-0.5\log (0.5)-0.5\log (0.5\left)\right)-(-0.75\log (0.75)-0.25\log (0.25\left)\right).$$

So your intuition is correct: these give the same answer.