And so this generates arbitrarily simple agents whose observed behaviour can only be described as maximising a utility function for arbitrarily complex utility functions (depending on how long you run them).
I object to the claim that agents that act randomly can be made “arbitrarily simple”. Randomness is basically definitionally complicated!
Eh, this seems a bit nitpicky. It’s arbitrarily simple given a call to a randomness oracle, which in practice we can approximate pretty easily. And it’s “definitionally” easy to specify as well: “the function which, at each call, returns true with 50% likelihood and false otherwise.”
If you get an ‘external’ randomness oracle, then you could define the utility function pretty simply in terms of the outputs of the oracle.
If the agent has a pseudo-random number generator (PRNG) inside it, then I suppose I agree that you aren’t going to be able to give it a utility function that has the standard set of convergent instrumental goals, and PRNGs can be pretty short. (Well, some search algorithms are probably shorter, but I bet they have higher Kt complexity, which is probably a better measure for agents)
I object to the claim that agents that act randomly can be made “arbitrarily simple”. Randomness is basically definitionally complicated!
Eh, this seems a bit nitpicky. It’s arbitrarily simple given a call to a randomness oracle, which in practice we can approximate pretty easily. And it’s “definitionally” easy to specify as well: “the function which, at each call, returns true with 50% likelihood and false otherwise.”
If you get an ‘external’ randomness oracle, then you could define the utility function pretty simply in terms of the outputs of the oracle.
If the agent has a pseudo-random number generator (PRNG) inside it, then I suppose I agree that you aren’t going to be able to give it a utility function that has the standard set of convergent instrumental goals, and PRNGs can be pretty short. (Well, some search algorithms are probably shorter, but I bet they have higher Kt complexity, which is probably a better measure for agents)