Consider the Sphex wasp, doing the same thing in response to the same stimulus. Would you say that this is not an agent, or would you say that it is part of an agent, and that extended agent did search in a “world model” instantiated in the parts of the world inhabited by ancestral wasps?
At this point, if you allow “world model” to be literally anything with mutual information including other macroscopic situations in the world, and “search” to be any process that gives you information about outcomes, then yes, I think you can guarantee that, probabilistically, getting a specific outcome requires information about that outcome (no free lunch), which implies “search” on a “world model.” As for goals, we can just ignore the apparent goals of the Sphex wasp and define a “real” agent (evolution) to have a goal defined by whatever informative process was at work (survival).
I think I do want to make my agent-like architecture general enough to include evolution. However, there might be a spectrum of agent-like-ness such that you can’t get much more than Sphex behavior with just evolution (without having a mesa-optimizer in there)
I think you can guarantee that, probabilistically, getting a specific outcome requires information about that outcome (no free lunch), which implies “search” on a “world model.”
Yeah, but do you think you can make it feel more like a formal proof?
Consider the Sphex wasp, doing the same thing in response to the same stimulus. Would you say that this is not an agent, or would you say that it is part of an agent, and that extended agent did search in a “world model” instantiated in the parts of the world inhabited by ancestral wasps?
At this point, if you allow “world model” to be literally anything with mutual information including other macroscopic situations in the world, and “search” to be any process that gives you information about outcomes, then yes, I think you can guarantee that, probabilistically, getting a specific outcome requires information about that outcome (no free lunch), which implies “search” on a “world model.” As for goals, we can just ignore the apparent goals of the Sphex wasp and define a “real” agent (evolution) to have a goal defined by whatever informative process was at work (survival).
I think I do want to make my agent-like architecture general enough to include evolution. However, there might be a spectrum of agent-like-ness such that you can’t get much more than Sphex behavior with just evolution (without having a mesa-optimizer in there)
Yeah, but do you think you can make it feel more like a formal proof?