I… don’t get what this post accomplishes. How can we test shard theory in reality? What does shard theory predict in reality? What kinds of facts about RL agents are predicted by shard theory?
Hm. I don’t think I was entirely clear about what tests I was thinking of. Suppose we have some real world RL agent trying to solve a maze to find cheese, and we’d like to model it as a shard-theoretic agent.
We propose a number of shards this agent could have based on previous experience with the agent, and what their contexts should be. In this case, perhaps, we have a go-towards-cheese shard, and a move-to-upper-right-corner shard and a distance-from-cheese context or possibly a time-since-start context.
An example of results which would disprove this instance of the theory is if we find there is no consistent function from contexts to shard-bids which accurately predicts our agent’s behavior, or if the solutions we find don’t actually compress our observations well. For instance, if they just list every situation our agent could be in, and append a number at the end corresponding to the move the agent makes.
We may also find problems when porting the models over to their weight-representations.
Thanks for the spelling corrections, those have been fixed. Grammatical errors have remained.
OK, and a thing-that-prevents-you-from-saying-”ah yes, that happened because a shard bid on it”-about-any-arbitrary-behavior is that shards are supposed to be reasonably farsighted/standard expected-utility-maximizers?
I… don’t get what this post accomplishes. How can we test shard theory in reality? What does shard theory predict in reality? What kinds of facts about RL agents are predicted by shard theory?
Misc: [resolved]
Hm. I don’t think I was entirely clear about what tests I was thinking of. Suppose we have some real world RL agent trying to solve a maze to find cheese, and we’d like to model it as a shard-theoretic agent.
We propose a number of shards this agent could have based on previous experience with the agent, and what their contexts should be. In this case, perhaps, we have a go-towards-cheese shard, and a move-to-upper-right-corner shard and a distance-from-cheese context or possibly a time-since-start context.
An example of results which would disprove this instance of the theory is if we find there is no consistent function from contexts to shard-bids which accurately predicts our agent’s behavior, or if the solutions we find don’t actually compress our observations well. For instance, if they just list every situation our agent could be in, and append a number at the end corresponding to the move the agent makes.
We may also find problems when porting the models over to their weight-representations.
Thanks for the spelling corrections, those have been fixed. Grammatical errors have remained.
OK, and a thing-that-prevents-you-from-saying-”ah yes, that happened because a shard bid on it”-about-any-arbitrary-behavior is that shards are supposed to be reasonably farsighted/standard expected-utility-maximizers?
Basically, yes.