TurnTrout comments on Goodhart’s Law in Reinforcement Learning

[TurnTrout]

We can recover a policy $\pi$ from its occupancy measure ${\eta }^{\pi }$ (up to the states that were never visited).

Actually, this claim in the original post is false for state-based occupancy measures, but it might be true for state-action measures. From p163 of On Avoiding Power-Seeking by Artificial Intelligence:

(Edited to qualify the counterexample as only applying to the state-based case)

[jacek]

Thanks for the comment! Note that we use state-action visitation distribution, so we consider trajectories that contain actions as well. This makes it possible to invert $\eta$ (as long as all states are visited). Using only states trajectories, it would indeed be impossible to recover the policy.

[TurnTrout]

Thanks, this was an oversight on my part.