Ethan (EJ) Watkins

• Ethan (EJ) Watkins 12 Jun 2023 8:04 UTC

  1 point

  0

  on: Six (and a half) in­tu­itions for KL divergence

  By the law of large numbers, $\frac{1}{N}{\sum }_{i=1}^N\ln Q_{\theta }\left(x_i\right)\to {\sum }_{x}P\left(x\right)\ln Q_{\theta }\left(x\right)$ almost surely. This is the cross entropy of $P$ and $Q_{\theta }$. Also note that if we subtract this from the entropy of $P$, we get $D_{KL}\left(P||Q_{\theta }\right)$. So minimising the cross entropy over $\theta$ is equivalent to maximising $D_{KL}\left(P||Q_{\theta }\right)$.

  I think the cross entropy of $P$ and $Q_{\theta }$ is actually $H(P,Q_{\theta })=-{\sum }_{x}P\left(x\right)\ln Q_{\theta }\left(x\right)$ (note the negative sign). The entropy of $P$ is $H\left(P\right)=-{\sum }_{x}P\left(x\right)\ln P\left(x\right)$. Since $D_{KL}\left(P||Q_{\theta }\right)={\sum }_{x}P\left(x\right)\left(\ln \right(P\left(x\right)-\ln Q_{\theta }\left(x\right))={\sum }_{x}P(x\left)\ln P\right(x)-{\sum }_{x}P(x)\ln Q_{\theta }\left(x\right)=-H(P)+H(P,Q_{\theta })$then the KL divergence is actually the cross entropy minus the entropy, not the other way around. So minimising the cross entropy over $\theta$ will minimise (not maximise) the KL divergence.
  
  I believe the next paragraph is still correct: the maximum likelihood estimator ${\theta }^{\ast }$ is the parameter which maximises $L({\hat{P}}_n;Q_{\theta })$, which minimises the cross-entropy, which minimises the KL divergence.
  
  Apologies if any of what I’ve said above is incorrect, I’m not an expert on this.

• Ethan (EJ) Watkins 1 Jun 2023 2:04 UTC

  1 point

  0

  on: Six (and a half) in­tu­itions for KL divergence

  $$D_{KL}\left(P||Q\right)=\sum _x{p}_x(\ln p_x-\ln q_x)=E\left[I_P\right(X)-I_Q(X\left)\right]$$

  I think there is a mistake in this equation.$I_P\left(X\right)$ and $I_Q\left(X\right)$ are the wrong way round. It should be:
  $$D_{KL}\left(P||Q\right)=\sum _x{p}_x(\ln p_x-\ln q_x)=E\left[I_Q\right(X)-I_P(X\left)\right]$$