A while back I was looking for toy examples of environments with different amounts of ‘naturalness’ to their abstractions, and along the way noticed a connection between this version of Gooder Regulator and the Blackwell order.
Inspired by this, I expanded on this perspective of preferences-over-models / abstraction a fair bit here.
It includes among other things:
the full preorder of preferences-shared-by-all-agents over maps X→M(vs. just the maximum)
an argument that actually we want to generalize to this diagram instead[1] :
an extension to preferences shared by ‘most’ agents based on Turner’s power-seeking theorems (and a formal connection to power-seeking)
Personally I think these results are pretty neat; I hope they might be of interest.
I also make one other argument there that directly responds to this post, re: the claim:
[The] “model” M definitely has to store exactly the posterior.
I think this sort of true, depending how you interpret it, but really it’s more accurate to say the whole regulator X→M→R has to encode the posterior, not just X→M.
Specifically, as you show, M(X) and (s↦P[S=s|X]) must induce the same equivalence classes of x. This means that they are isomorphic as functions in a particular sense.
But it turns out: lots of (meaningfully) different systems X→S can lead to the same optimal model M(x). This means you can’t recover the full system transition function just by looking at M(x); so it doesn’t really store the whole posterior.
On the other hand, you can uniquely recover the whole posterior (up to technicalities) from M(x) plus the agent’s optimal strategies π:(M,Y)→ΔR. So it is still fair to say the agent as a whole most model the Bayesian posterior; but I’d say it’s not just the model M(X) which does it.
Which in this perspective basically means allowing the ‘system’ or ‘world’ we study to include restrictions on the outcome mapping, e.g. that (s,a), and (s′,a′) lead to the same outcome, which must be given a common utility by any given game. Looking back at this again, I’m not sure I described the distinction quite right in my post (since this setting as you gave it already has a Z distinct from u(Z)), but there is still a distinction.
A while back I was looking for toy examples of environments with different amounts of ‘naturalness’ to their abstractions, and along the way noticed a connection between this version of Gooder Regulator and the Blackwell order.
Inspired by this, I expanded on this perspective of preferences-over-models / abstraction a fair bit here.
It includes among other things:
the full preorder of preferences-shared-by-all-agents over maps X→M(vs. just the maximum)
an argument that actually we want to generalize to this diagram instead[1] :
an extension to preferences shared by ‘most’ agents based on Turner’s power-seeking theorems (and a formal connection to power-seeking)
Personally I think these results are pretty neat; I hope they might be of interest.
I also make one other argument there that directly responds to this post, re: the claim:
I think this sort of true, depending how you interpret it, but really it’s more accurate to say the whole regulator X→M→R has to encode the posterior, not just X→M.
Specifically, as you show, M(X) and (s↦P[S=s|X]) must induce the same equivalence classes of x. This means that they are isomorphic as functions in a particular sense.
But it turns out: lots of (meaningfully) different systems X→S can lead to the same optimal model M(x). This means you can’t recover the full system transition function just by looking at M(x); so it doesn’t really store the whole posterior.
On the other hand, you can uniquely recover the whole posterior (up to technicalities) from M(x) plus the agent’s optimal strategies π:(M,Y)→ΔR. So it is still fair to say the agent as a whole most model the Bayesian posterior; but I’d say it’s not just the model M(X) which does it.
Which in this perspective basically means allowing the ‘system’ or ‘world’ we study to include restrictions on the outcome mapping, e.g. that (s,a), and (s′,a′) lead to the same outcome, which must be given a common utility by any given game. Looking back at this again, I’m not sure I described the distinction quite right in my post (since this setting as you gave it already has a Z distinct from u(Z)), but there is still a distinction.