Main difference between what they’re doing and what I’m doing: they’re using explicit utility & maximization nodes; I’m not. It may be that this doesn’t actually matter. The representation I’m using certainly allows for utility maximization—a node downstream of a cloud can just be a maximizer for some utility on the nodes of the cloud-model. The converse question is less obvious: can any node downstream of a cloud be represented by a utility maximizer (with a very artificial “utility”)? I’ll probably play around with that a bit; if it works, I’d be able to re-use the equivalence results in that paper. If it doesn’t work, then that would demonstrate a clear qualitative difference between “goal-directed” behavior and arbitrary behavior in these sorts of systems, which would in turn be useful for alignment—it would show a broad class of problems where utility functions do constrain.
Another thing you might find useful is Dennett’s discussion of what an agent is (see first few chapters of Bacteria to Bach). Basically, he argues that an agent is something we ascribe beliefs and goals to. If he’s right, then an agent should basically always have a utility function.
Your post focuses on the belief part, which is perhaps the more interesting aspect when thinking about strange loops and similar.
There is a paper which I believe is trying to do something similar to what you are attempting here:
Networks of Influence Diagrams: A Formalism for Representing Agents’ Beliefs and Decision-Making Processes, Gal and Pfeffer, Journal of Artificial Intelligence Research 33 (2008) 109-147
Are you aware of it? How do you think their ideas relate to yours?
Very interesting, thank you for the link!
Main difference between what they’re doing and what I’m doing: they’re using explicit utility & maximization nodes; I’m not. It may be that this doesn’t actually matter. The representation I’m using certainly allows for utility maximization—a node downstream of a cloud can just be a maximizer for some utility on the nodes of the cloud-model. The converse question is less obvious: can any node downstream of a cloud be represented by a utility maximizer (with a very artificial “utility”)? I’ll probably play around with that a bit; if it works, I’d be able to re-use the equivalence results in that paper. If it doesn’t work, then that would demonstrate a clear qualitative difference between “goal-directed” behavior and arbitrary behavior in these sorts of systems, which would in turn be useful for alignment—it would show a broad class of problems where utility functions do constrain.
Glad you liked it.
Another thing you might find useful is Dennett’s discussion of what an agent is (see first few chapters of Bacteria to Bach). Basically, he argues that an agent is something we ascribe beliefs and goals to. If he’s right, then an agent should basically always have a utility function.
Your post focuses on the belief part, which is perhaps the more interesting aspect when thinking about strange loops and similar.