I’ve been thinking today about problems of self-reference in agent foundations. I wanted to share a toy model I came up with for the emergence of coherent identity in an agent (of course, for values of those words which make them correspond to the things I’m about to define, so caveat emptor). Roughly, factoring the world into (agent) x (environment) induces a very natural way to divide patterns of behaviour into so-called ‘motivational orbits’, and particular such orbit is selected for in the long-time limit. The math maps cleanly onto natural selection acting between these ‘motivational orbits’ - thus they are a very natural way to divide an ‘agent’ into competing subagents. Further, there is a natural way to assign a value-function-like object to each of these subagents; the long-term-selected orbit then has a single coherent value-function.
As ever, I’m more motivated by the mathematical archetype and less with having the model produce a particular phenomenon—its one of those things which I think would make a good starting point for an ontology in a more complete theory. Other than the standard RL terminology, you can look at quasispecies for the inspiration to this model.
Example in words:
Bob is a good Calvinist who tends to wake up energized at 4am, work until 7pm, and then eat a plain dinner and pray until bedtime at 8pm. One day, he sleeps very poorly because of a thunderstorm or something—he wakes up ill-rested and demoralized, watches TV for a bit, and generally hangs around for the rest of the day. He sleeps well, and is ‘back at it’ the next day.
Bob as an entity may want or feel different things throughout this process, both on his good days and his bad days. He may want to work, or to watch TV, or to pray. However there is a sense in which the ‘routine’ days are more ‘stable’, and to the extent that he regularly returns to his routine after some perturbation, we might be justified in considering them more ‘representative’ of Bob as a person. For our purposes “Bob on a good day” and “Bob on a bad day” are subagents, and they ‘compete’ to be expressed more frequently (i.e., over a long enough period of time, we see more of the one which is more stable).
Below I present a model where something like certain ‘routines’ are the natural joints at which to carve the world; so carved, these ‘routines’ undergo a natural selection process until only the dominant one is left. I present this as a toy model for thinking about the emergence of a coherent identity. I think its worth considering because the quasispecies idea is a cool-and-maybe-useful joint-carving motif.
We consider a markov chain (not an MDP!) defined over the cross product (internal state, external state), call these and s, respectively. The agent state determines its policy . The traditional transition function on the external world is . We introduce also the agent-state transition matrix . This defines a Markov process on the joint state , with evolution operator . can be eigendecomposed into . (where the inner product associated to is with whatever normalization makes this work). Then for some initial distribution , its time evolution is :
These are the ‘behavioural orbits’ I was talking about—because they are ‘stable’ - whereas generic evolves in some whacky way, evolves into itself. Moreover, Perron-Frobenius theorem (under the standard conditions) tells us that the largest eigenvalue , and that (generically), all other eigenvalues are less than 1.
So if you as an external observer have some initial prior over the agent-environment joint state , after a bit of a wait (and without making any aditional observations...), you can safely update to thinking the joint state follows the distribution, call it .
If you want to go a bit further and predict the agent’s internal state, you can take this distribution and condition on .
Worth noting we can also define, for any prior , we can define the implied policy an external observer would see:
GPT-5.5 also claims you can fiddle and multiply the internal-state transition function with a weight which allows you to get something close to a soft Bellman update, with value-function - perhaps a bit cringe / a stretch, but this is a way you could define a at-least-kinda-not-tautological “value function”—it’s kinda cute because each sub-agent / eigenmode has its own value function , and only in ‘equilibrium’ does the system as a whole “have a value function”. This is a nice bridge between absence of subagent-conflict and coherence as an agent. In principle, if you don’t know, or don’t want to posit the eigenmodes, you could probably use this to think about how agents might have multiple active value functions.
I’ve been thinking today about problems of self-reference in agent foundations. I wanted to share a toy model I came up with for the emergence of coherent identity in an agent (of course, for values of those words which make them correspond to the things I’m about to define, so caveat emptor). Roughly, factoring the world into (agent) x (environment) induces a very natural way to divide patterns of behaviour into so-called ‘motivational orbits’, and particular such orbit is selected for in the long-time limit. The math maps cleanly onto natural selection acting between these ‘motivational orbits’ - thus they are a very natural way to divide an ‘agent’ into competing subagents. Further, there is a natural way to assign a value-function-like object to each of these subagents; the long-term-selected orbit then has a single coherent value-function.
As ever, I’m more motivated by the mathematical archetype and less with having the model produce a particular phenomenon—its one of those things which I think would make a good starting point for an ontology in a more complete theory. Other than the standard RL terminology, you can look at quasispecies for the inspiration to this model.
Example in words: Bob is a good Calvinist who tends to wake up energized at 4am, work until 7pm, and then eat a plain dinner and pray until bedtime at 8pm. One day, he sleeps very poorly because of a thunderstorm or something—he wakes up ill-rested and demoralized, watches TV for a bit, and generally hangs around for the rest of the day. He sleeps well, and is ‘back at it’ the next day.
Bob as an entity may want or feel different things throughout this process, both on his good days and his bad days. He may want to work, or to watch TV, or to pray. However there is a sense in which the ‘routine’ days are more ‘stable’, and to the extent that he regularly returns to his routine after some perturbation, we might be justified in considering them more ‘representative’ of Bob as a person. For our purposes “Bob on a good day” and “Bob on a bad day” are subagents, and they ‘compete’ to be expressed more frequently (i.e., over a long enough period of time, we see more of the one which is more stable).
Below I present a model where something like certain ‘routines’ are the natural joints at which to carve the world; so carved, these ‘routines’ undergo a natural selection process until only the dominant one is left. I present this as a toy model for thinking about the emergence of a coherent identity. I think its worth considering because the quasispecies idea is a cool-and-maybe-useful joint-carving motif.
We consider a markov chain (not an MDP!) defined over the cross product (internal state, external state), call these
These
So if you as an external observer have some initial prior over the agent-environment joint state
If you want to go a bit further and predict the agent’s internal state, you can take this distribution and condition on
Worth noting we can also define, for any prior
GPT-5.5 also claims you can fiddle and multiply the internal-state transition function