I like this framework, but I think it’s still a bit tricky about how to draw lines around agents/optimization processes.
For instance, I can think of ways to make a rock interact with far away variables by e.g., coupling it to a human who presses various buttons based on the internal state or the rock. In this case, would you draw the boundary around both the rock and the human and say that that unit is “optimizing”?
That seems a bit weird, given that the human is clearly the “optimizer” in this scenario. And drawing a line around only the rock or only the human seems wrong too (human is clearly using the rock to do this strange optimization process and rock is relying on the human for this to occur). Curious about your thoughts.
Also, I’m not sure that agents always optimize things far away from themselves. Bacteria follow chemical gradients (and this feels agent-y to me), but the chemicals are immediately present both temporally and spatially. There is some sense in which bacteria are “trying” to get somewhere far away (the maximum concentration), but they’re also pretty locally achieving the goal, i.e., the actions they take in the present are very close in space and time to what they’re trying to achieve (eat the chemicals).
Bacteria follow chemical gradients (and this feels agent-y to me), but the chemicals are immediately present both temporally and spatially.
Subtle point here: most of the agenty things which persist over time (like humans or bacteria) are optimizing their own future state, and it’s that future state which is far away in space/time from their current decision.
For instance, I can think of ways to make a rock interact with far away variables by e.g., coupling it to a human who presses various buttons based on the internal state or the rock. In this case, would you draw the boundary around both the rock and the human and say that that unit is “optimizing”?
The real answer here is that this post isn’t meant to handle that question. Some boundaries are clearly more natural optimizer boundaries than others, but this post is not yet trying to fully say which, it’s just laying some groundwork/necessary conditions. One of the necessary conditions which this post does not address is robustness of the optimization to changes in the environment, which is what makes e.g. the rock look like it’s not an optimizer.
I like this framework, but I think it’s still a bit tricky about how to draw lines around agents/optimization processes.
For instance, I can think of ways to make a rock interact with far away variables by e.g., coupling it to a human who presses various buttons based on the internal state or the rock. In this case, would you draw the boundary around both the rock and the human and say that that unit is “optimizing”?
That seems a bit weird, given that the human is clearly the “optimizer” in this scenario. And drawing a line around only the rock or only the human seems wrong too (human is clearly using the rock to do this strange optimization process and rock is relying on the human for this to occur). Curious about your thoughts.
Also, I’m not sure that agents always optimize things far away from themselves. Bacteria follow chemical gradients (and this feels agent-y to me), but the chemicals are immediately present both temporally and spatially. There is some sense in which bacteria are “trying” to get somewhere far away (the maximum concentration), but they’re also pretty locally achieving the goal, i.e., the actions they take in the present are very close in space and time to what they’re trying to achieve (eat the chemicals).
Subtle point here: most of the agenty things which persist over time (like humans or bacteria) are optimizing their own future state, and it’s that future state which is far away in space/time from their current decision.
The real answer here is that this post isn’t meant to handle that question. Some boundaries are clearly more natural optimizer boundaries than others, but this post is not yet trying to fully say which, it’s just laying some groundwork/necessary conditions. One of the necessary conditions which this post does not address is robustness of the optimization to changes in the environment, which is what makes e.g. the rock look like it’s not an optimizer.