Naively selecting from the maximum entropy distribution (as narrowed by all the conditions the predictor is aware of) still permits the model to collapse reflective predictions in a way that permits internally motivated goal-directed behavior (leaving aside whether it’s probable), because it’s aware of the reflective nature of the prediction.
Hm, isn’t if we apply maximum entropy principle universally, aren’t we also obliged to apply it reflectively, i.e., model oneself as a maximum-entropy (active inference) agent? BTW, this is exactly the setup explored by Ramstead et al. (2022).
In other words, to get to what I mean by “minimally collapsed,” there seems to be some additional counterfactual surgery required. For example, the model could output the distribution that it would output if it knew it did not influence the prediction.
Looks more like a suitable inductive bias and/or bias is needed rather than causal surgery.
Hm, isn’t if we apply maximum entropy principle universally, aren’t we also obliged to apply it reflectively, i.e., model oneself as a maximum-entropy (active inference) agent?
If you precisely define what it means to apply it “universally” such that it gets you the desired behavior, sure. And to be clear, I’m not saying that’s a hard/impossible problem or anything like that, it’s just not directly implied by all things which match the description “follows the principle of maximum entropy.”
Looks more like a suitable inductive bias and/or bias is needed rather than causal surgery.
If you were actually trying to implement this, yes, I wouldn’t recommend routing through weird counterfactuals. (I just bring those up as a way of describing the target behavior.)
In fact, because even the version I outlined in the added footnote can still suffer from collapse in the case of convergent acausal strategies across possible predictors, I would indeed strongly recommend pushing for some additional bias that gives you more control over how the distribution looks. I think that’s pretty tractable, too.
Hm, isn’t if we apply maximum entropy principle universally, aren’t we also obliged to apply it reflectively, i.e., model oneself as a maximum-entropy (active inference) agent? BTW, this is exactly the setup explored by Ramstead et al. (2022).
Looks more like a suitable inductive bias and/or bias is needed rather than causal surgery.
If you precisely define what it means to apply it “universally” such that it gets you the desired behavior, sure. And to be clear, I’m not saying that’s a hard/impossible problem or anything like that, it’s just not directly implied by all things which match the description “follows the principle of maximum entropy.”
If you were actually trying to implement this, yes, I wouldn’t recommend routing through weird counterfactuals. (I just bring those up as a way of describing the target behavior.)
In fact, because even the version I outlined in the added footnote can still suffer from collapse in the case of convergent acausal strategies across possible predictors, I would indeed strongly recommend pushing for some additional bias that gives you more control over how the distribution looks. I think that’s pretty tractable, too.