Bit of an accidental pun, here. “Integrating additional information” (in the usual sense of the phrase), has exactly the opposite meaning of “integrate out a variable”—when we integrate over the variable (in the mathy sense of the phrase), we’re throwing out whatever information it contains.
So, yes—it does mean that we can’t expect an approximation to improve when we integrate in additional information (in the layman’s sense of the phrase).
Doesn’t
D_KL(P[X]||Q[X])≤E_Y[D_KL(P[X|Y]||Q[X|Y])]
mean that you can’t expect to improve by integrating in additional information Y?
Bit of an accidental pun, here. “Integrating additional information” (in the usual sense of the phrase), has exactly the opposite meaning of “integrate out a variable”—when we integrate over the variable (in the mathy sense of the phrase), we’re throwing out whatever information it contains.
So, yes—it does mean that we can’t expect an approximation to improve when we integrate in additional information (in the layman’s sense of the phrase).