Thane Ruthenis comments on Utility functions and probabilities are entangled

[Thane Ruthenis]

You have to have some information besides behavior alone, or you’ll have this extra degree of freedom in making interpretations.

Yup. Seems to be an instance of the general principle that black-box agents can be assigned any values whatsoever.