Maximum redund=Maximal redund, Minimum mediator=Minimal mediator, & Naturality=Shrinking regimes of tradeoffs
Previously, we’ve shown that given constraints on the entropy of a natural latent variable Λ, the mediation and redundancy errors are minimized exactly when the sum of mutual information with observables ∑iI(Xi;Λ) is maximized. In addition, the entropy H(Λ) of the latent variable is exactly the parameter that represents the tradeoff between the mediation and redundancy condition. In particular, the mediation error can only reduce with H(Λ) while the redundancy errors can only increase with H(Λ).
However, there may be regimes where changes in H(Λ) can reduce the mediation error without increasing the redundancy errors or vice versa. For instance:
If we gradually increase H(Λ) while perfectly preserving the redundancy conditions H(Λ|Xi)=0,∀i, then we can reduce the mediation error (as it can only reduce with increasing H(Λ)) without increasing the redundancy errors (as they stay 0). Increasing H(Λ) therefore becomes a weak pareto-improvement over the naturality conditions
Similarly, if we gradually reduce H(Λ) while perfectly preserving the mediation condition TC(X|Λ)=0, then we can reduce the redundancy errors without increasing the mediation error (as it stays 0)
If we define a maximum redund Λ∗ as a latent variable that satisfies the redundancy conditions H(Λ∗|Xi)=0 and has the maximum entropy among redunds, then H(Λ)≤H(Λ∗) represents the regime where we can increase H(Λ) without increasing the redundancy errors, since increasing H(Λ) beyond H(Λ∗) would necessarily violate the redundancy condition given our assumption of maximum entropy
Similarly, define a minimum mediator Λ∗ as a mediator with minimal entropy (among mediators). Then H(Λ)≥H(Λ∗) represents the regime where we can reduce entropy without increasing the mediation error, since reducing H(Λ) below H(Λ∗) necessarily violates the mediation condition.
Combining these ideas, H(Λ∗)≤H(Λ)≤H(Λ∗) represents the regime where changing H(Λ) actually presents a tradeoff between the mediation and redundancy errors; the minimum mediator and maximum redund marks the boundaries for when weak pareto-improvements are possible.
Maximal redunds and minimal mediators
In natural latents we care about the uniqueness of latent variables, which is why we have concepts like minimal mediators and maximal redunds:
A minimal mediator is a mediator Λ such that for any other mediator Λ′ we have H(Λ|Λ′)<ϵ. So a minimal mediator is an approximately deterministic function of any other mediator
A maximal redund is a redund Λ such that for any other redund Λ′ we have H(Λ′|Λ)<ϵ. So any redund is approximately a deterministic function of the maximal redund
Through a universal-property-flavored proof, we can show approximate isomorphism among any pair of minimal mediators X,Y: Since X is a minimal mediator and Y is a mediator, Y approximately determines X, and using a similar argument we conclude X determines Y. The same reasoning also allows us to derive uniqueness of any pair of maximal redunds. Naturality occurs when the maximal redund converge with the minimal mediator.
However, note that the concepts of minimal mediators and maximal redunds are at least conceptually distinct from minimum mediators and maximum redunds. We shall therefore prove that these concepts are mathematically equivalent. This can be useful because it’s much easier to find minimum mediators and maximum redunds computationally, but we ultimately care about the unqiueness property offered by minimal mediators and maximal redunds, proving an equivalence enables the former to have the uniqueness guarantees of the latter.
Let Y be a minimal mediator and Z be a minimum mediator
Since Y is a minimal mediator and Z is a mediator, we have H(Y|Z)<ϵ which means I(Y;Z)≥H(Y)−ϵ
H(Z|Y)=H(Z)−I(Z;Y)≤H(Z)−H(Y)+ϵ
Since Z has minimal entropy, we have H(Z)−H(Y)<0 which means H(Z|Y)<ϵ
For any other latent Λ, we have H(Z|Λ)≤H(Z|Y)+H(Y|Λ)<2ϵ, which means Z is also a minimal mediator (up to error 2ϵ)
In addition, we have H(Y)≤H(Y,Z)=H(Z)+H(Y|Z)≤H(Z)+ϵ so Y is also an approximate minimum mediator
Maximum redund= Maximal redund
Proof:
Suppose that Z is maximum redund of X1…Xn and Y is any other redund
(Z,Y) is a redund since both Z and Y are deterministic functions of any Xi, since Z is maximum redund we have H(Z)≥H(Z,Y)
H(Z,Y)=H(Z)+H(Y|Z) hence we have H(Z)≥H(Z)+H(Y|Z). Since H(Y|Z) is nonegative, we must have H(Y|Z)=0
As a result, H(Z) is also a maximal redund
Similarly, suppose that Y is a maximal redund, then H(Z)≤H(Z,Y)=H(Y)+H(Z|Y)≤H(Y)+ϵ, which means H(Y)≥H(Z)−ϵ and Y is also an approximate maximum redund.
Naturality as shrinking regime of tradeoffs
Recall that H(Λ∗)≤H(Λ)≤H(Λ∗) (where Λ∗ is the minimum mediator and Λ∗ is the maximum redund) represents the regime where changing H(Λ) actually presents a tradeoff between the mediation and redundancy errors. Due to the equivalence we proved, we can also think of Λ∗ as the minimal mediator and Λ∗ as the maximal redund.
We also know that naturality occurs when the minimal mediator converges with the maximal redund (as a natural latent satisfies both mediation and redundancy, and mediator determines redund); we can picture this convergence as if we’re shrinking the gap between H(Λ∗) and H(Λ∗). In other words, naturality occurs exactly when the regime of tradeoff (H(Λ∗)≤H(Λ)≤H(Λ∗)) between the redundancy and mediation error is small. If we have exact naturality H(Λ∗)=H(Λ∗), then pareto-improvements on the naturality conditions can always be made by nudging H(Λ) closer to H(Λ∗).
Combining this with our previous result, we conclude that that maximizing ∑iI(Xi,Λ) represents strong pareto-improvements over the naturality conditions; H(Λ)≤H(Λ∗) and H(Λ)≥H(Λ∗) represents the regime where we can have weak pareto-improvements by nudging H(Λ) closer to the boundary of H(Λ∗) or H(Λ∗); whereas H(Λ∗)≤H(Λ)≤H(Λ∗) represents the regime of real tradeoffs between naturality conditions. An approximate natural latent exist exactly when the regime of real tradeoffs is small and we can pareto-improve towards naturality
Maximum redund=Maximal redund, Minimum mediator=Minimal mediator, & Naturality=Shrinking regimes of tradeoffs
Previously, we’ve shown that given constraints on the entropy of a natural latent variable Λ, the mediation and redundancy errors are minimized exactly when the sum of mutual information with observables ∑iI(Xi;Λ) is maximized. In addition, the entropy H(Λ) of the latent variable is exactly the parameter that represents the tradeoff between the mediation and redundancy condition. In particular, the mediation error can only reduce with H(Λ) while the redundancy errors can only increase with H(Λ).
However, there may be regimes where changes in H(Λ) can reduce the mediation error without increasing the redundancy errors or vice versa. For instance:
If we gradually increase H(Λ) while perfectly preserving the redundancy conditions H(Λ|Xi)=0,∀i, then we can reduce the mediation error (as it can only reduce with increasing H(Λ)) without increasing the redundancy errors (as they stay 0). Increasing H(Λ) therefore becomes a weak pareto-improvement over the naturality conditions
Similarly, if we gradually reduce H(Λ) while perfectly preserving the mediation condition TC(X|Λ)=0, then we can reduce the redundancy errors without increasing the mediation error (as it stays 0)
If we define a maximum redund Λ∗ as a latent variable that satisfies the redundancy conditions H(Λ∗|Xi)=0 and has the maximum entropy among redunds, then H(Λ)≤H(Λ∗) represents the regime where we can increase H(Λ) without increasing the redundancy errors, since increasing H(Λ) beyond H(Λ∗) would necessarily violate the redundancy condition given our assumption of maximum entropy
Similarly, define a minimum mediator Λ∗ as a mediator with minimal entropy (among mediators). Then H(Λ)≥H(Λ∗) represents the regime where we can reduce entropy without increasing the mediation error, since reducing H(Λ) below H(Λ∗) necessarily violates the mediation condition.
Combining these ideas, H(Λ∗)≤H(Λ)≤H(Λ∗) represents the regime where changing H(Λ) actually presents a tradeoff between the mediation and redundancy errors; the minimum mediator and maximum redund marks the boundaries for when weak pareto-improvements are possible.
Maximal redunds and minimal mediators
In natural latents we care about the uniqueness of latent variables, which is why we have concepts like minimal mediators and maximal redunds:
A minimal mediator is a mediator Λ such that for any other mediator Λ′ we have H(Λ|Λ′)<ϵ. So a minimal mediator is an approximately deterministic function of any other mediator
A maximal redund is a redund Λ such that for any other redund Λ′ we have H(Λ′|Λ)<ϵ. So any redund is approximately a deterministic function of the maximal redund
Through a universal-property-flavored proof, we can show approximate isomorphism among any pair of minimal mediators X,Y: Since X is a minimal mediator and Y is a mediator, Y approximately determines X, and using a similar argument we conclude X determines Y. The same reasoning also allows us to derive uniqueness of any pair of maximal redunds. Naturality occurs when the maximal redund converge with the minimal mediator.
However, note that the concepts of minimal mediators and maximal redunds are at least conceptually distinct from minimum mediators and maximum redunds. We shall therefore prove that these concepts are mathematically equivalent. This can be useful because it’s much easier to find minimum mediators and maximum redunds computationally, but we ultimately care about the unqiueness property offered by minimal mediators and maximal redunds, proving an equivalence enables the former to have the uniqueness guarantees of the latter.
Minimum mediator = Minimal mediator (when minimal mediator exists)
Proof:
Let Y be a minimal mediator and Z be a minimum mediator
Since Y is a minimal mediator and Z is a mediator, we have H(Y|Z)<ϵ which means I(Y;Z)≥H(Y)−ϵ
Since Z has minimal entropy, we have H(Z)−H(Y)<0 which means H(Z|Y)<ϵ
For any other latent Λ, we have H(Z|Λ)≤H(Z|Y)+H(Y|Λ)<2ϵ, which means Z is also a minimal mediator (up to error 2ϵ)
In addition, we have H(Y)≤H(Y,Z)=H(Z)+H(Y|Z)≤H(Z)+ϵ so Y is also an approximate minimum mediator
Maximum redund= Maximal redund
Proof:
Suppose that Z is maximum redund of X1…Xn and Y is any other redund
H(Z,Y)=H(Z)+H(Y|Z) hence we have H(Z)≥H(Z)+H(Y|Z). Since H(Y|Z) is nonegative, we must have H(Y|Z)=0
As a result, H(Z) is also a maximal redund
Similarly, suppose that Y is a maximal redund, then H(Z)≤H(Z,Y)=H(Y)+H(Z|Y)≤H(Y)+ϵ, which means H(Y)≥H(Z)−ϵ and Y is also an approximate maximum redund.
Naturality as shrinking regime of tradeoffs
Recall that H(Λ∗)≤H(Λ)≤H(Λ∗) (where Λ∗ is the minimum mediator and Λ∗ is the maximum redund) represents the regime where changing H(Λ) actually presents a tradeoff between the mediation and redundancy errors. Due to the equivalence we proved, we can also think of Λ∗ as the minimal mediator and Λ∗ as the maximal redund.
We also know that naturality occurs when the minimal mediator converges with the maximal redund (as a natural latent satisfies both mediation and redundancy, and mediator determines redund); we can picture this convergence as if we’re shrinking the gap between H(Λ∗) and H(Λ∗). In other words, naturality occurs exactly when the regime of tradeoff (H(Λ∗)≤H(Λ)≤H(Λ∗)) between the redundancy and mediation error is small. If we have exact naturality H(Λ∗)=H(Λ∗), then pareto-improvements on the naturality conditions can always be made by nudging H(Λ) closer to H(Λ∗).
Combining this with our previous result, we conclude that that maximizing ∑iI(Xi,Λ) represents strong pareto-improvements over the naturality conditions; H(Λ)≤H(Λ∗) and H(Λ)≥H(Λ∗) represents the regime where we can have weak pareto-improvements by nudging H(Λ) closer to the boundary of H(Λ∗) or H(Λ∗); whereas H(Λ∗)≤H(Λ)≤H(Λ∗) represents the regime of real tradeoffs between naturality conditions. An approximate natural latent exist exactly when the regime of real tradeoffs is small and we can pareto-improve towards naturality