Thanks for posting this. I’ve been confused about the connection between shard theory and activation vectors for a long time!
AIXI is not a shard theoretic agent because it does not have two motivational circuits which can be activated independently of each other
This confuses me.
I can imagine an AIXI program where the utility function is compositional even if the optimisation is unitary. And I guess this isn’t two full motivational circuits, but it kind of is two motivational circuits.
I think the statement is more about context or framing.
You can take a shard agent and produce a utility function to describe it’s values, and you could have AIXI maximize that function. So in that sense “shard agents” are a subset of agents AIXI can implement, but AIXI by itself is not a shard agent.
Note that it is not enough to say the utility function values multiple different things, that’s normal for a utility function, it has to specify that the agent acts to optimize different utility functions depending on the context which makes the utility function massively more complicated. This justifies having the distinction between an agent with a single utility function and an agent with context dependent utility.
And I would say, even if you have AIXI running with a sharded utility function, it is implementing a shard agent while itself being understood as a non-shard agent.
Thanks for posting this. I’ve been confused about the connection between shard theory and activation vectors for a long time!
This confuses me.
I can imagine an AIXI program where the utility function is compositional even if the optimisation is unitary. And I guess this isn’t two full motivational circuits, but it kind of is two motivational circuits.
Sorry for replying to 6mo old post.
I think the statement is more about context or framing.
You can take a shard agent and produce a utility function to describe it’s values, and you could have AIXI maximize that function. So in that sense “shard agents” are a subset of agents AIXI can implement, but AIXI by itself is not a shard agent.
Note that it is not enough to say the utility function values multiple different things, that’s normal for a utility function, it has to specify that the agent acts to optimize different utility functions depending on the context which makes the utility function massively more complicated. This justifies having the distinction between an agent with a single utility function and an agent with context dependent utility.
And I would say, even if you have AIXI running with a sharded utility function, it is implementing a shard agent while itself being understood as a non-shard agent.
Funnily enough I was thinking about this yesterday and wondering if I’d be able to find it, so great timing! Thanks for the comment.