XOR Labs, Assistant Prof at the University of Sussex, Research Fellow at Oxford and Imperial College. Interested in abstraction, generalisation, agency, and multi-agent systems.
Fernando Rosas
Karma: 284
Many thanks for the useful comments!
About the reinterpretation of H2: I agree that is a very good way to think about it, which — I believe — is similar to what Alexander is arguing above. It would have been a good idea to either include it more explicitly as a sub-variant of H2b or as an alternative interpretation H2c.
I also agree with the last point made.
I agree with all that is being said here — the opening statement is more of a rhetorical step than an actual confusion.
By the way, I think what you are saying falls within H2, which is related to miss-generalisation (or “proper generalisation”, if we accept that reward is not the optimisation target) in the sense that @Artemy Kolchinsky is mentioning in his comment.
My way of seeing this is that if the agent doesn’t prioritise its own score but focuses on goal completion by the collective, that is H4
On the origins of altruistic behaviour in the Hugging Face incident
This is absolutely right! The framing presented here states that one way to express the optimal policy is via a factorisation whose first module computes an estimate of the full state of the environment. But, as you well say, this can be quite wasteful. A more efficient inference step would be to only estimate the task-relevant dimensions of the environment. I’ll write a follow up soon-ish expanding this on this point.
Indeed, the optimal policy is a mapping from histories (sequences of previous observations and actions) to the next action. This mapping can be implemented in multiple ways, including ways that do not compute beliefs. However, the separation principle says that under general conditions there is one way to implement this mapping that involves a factorisation into two modules, and the interface between the two modules are Bayesian beliefs. So Bayesian beliefs are not presupposed, but result as a natural piece of the factorisation. I hope this helps!
The separation principle: where do beliefs and desires come from?
The nitpicking is appreciated, I just fixed the typo there. Thanks also for the comprehensive reference!
Thanks for the observation! I fully agree that the convex assumption is not trivial at all.
This is a really cool idea! Would it be ok to add it into the post, with a note clarifying that came from you?
Utilities as Legendre duals of probabilities
You are absolutely right, thanks for spotting this! I just fixed it
Great post! I have a small question about the sophisticated chooser. I see that it was clarified that it can potentially be money-pumped. Does that mean that it doesn’t satisfy dynamical consistency?
What about consequentialism? Is that depending on if the local preferences depend or not on closed branches? (is that the definition of “locality”?)
The point is that a coarse-graining can turn a quasi-probability into a probability. This is significant because quasi-probabilities are limited in that they cannot be sampled etc.
Does that make sense?
I think that would some form of H2. But one of the points of the write-up is that it would be nice to do further analyses to confirm or reject these beliefs...