Alex Gibson comments on $500 Bounty Problem: Are (Approximately) Deterministic Natural Latents All You Need?

[Alex Gibson]

I’m confused by this. The KL term we are looking at in the deterministic case is
$D_{\text{KL}}\left(P\right[X,\Lambda \left]||P\right[\Lambda \left]P\right[X_1|\Lambda \left]P\right[X_2|\Lambda \left]\right)$, right?

For simplicity, we imagine we have finite discrete spaces. Then this would blow up if $P[X=(x_1,x_2),\Lambda =\lambda ]\ne 0$, and $P[\Lambda =\lambda ]P[X_1=x_1|\Lambda =\lambda ]P[X_2=x_2|\Lambda =\lambda ]=0$. But this is impossible, because any of the terms in the product being 0 imply that $P[X=(x_1,x_2),\Lambda =\lambda ]$ is $0$.

Intuitively, we construct an optimal code for encoding the distribution $P\left[\Lambda \right]P\left[X_1|\Lambda \right]P\left[X_2|\Lambda \right]$, and the KL divergence measures how many more bits on average we need to encode a message than optimal, if the true distribution is given by $P[X,\Lambda ]$. Issues occur when but the true distribution $P[X,\Lambda ]$ takes on values which never occur according to $P\left[\Lambda \right]P\left[X_1|\Lambda \right]P\left[X_2|\Lambda \right]$, i.e: the optimal code doesn’t account for those values potentially occurring.

Potentially there are subtleties when we have continuous spaces. In any case I’d be grateful if you’re able to elaborate.

[johnswentworth]

Yeah, I’ve since updated that deterministic functions are probably the right thing here after all, and I was indeed wrong in exactly the way you’re pointing out.