I mean I also imagine that the agents which survive the best are the ones that are trying to survive. I don’t understand why we’d expect agents that are trying to survive and also accomplish some separate arbitrary infinite-horizon goal would outperform those that are just trying to maintain the conditions necessary for their survival without additional baggage.
To be clear, my position is not “homeostatic agents make good tools and so we should invest efforts in creating them”. My position is “it’s likely that homeostatic agents have significant competitive advantages against unbounded-horizon consequentialist ones, so I expect the future to be full of them, and expect quite a bit of value in figuring out how to make the best of that”.
Ah ok. I was responding to your post’s initial prompt: “I still don’t really intuitively grok why I should expect agents to become better approximated by “single-minded pursuit of a top-level goal” as they gain more capabilities.” (The reason to expect this is that “single-minded pursuit of a top-level goal,” if that goal is survival, could afford evolutionary advantages.)
But I agree entirely that it’d be valuable for us to invest in creating homeostatic agents. Further, I think calling into doubt western/capitalist/individualist notions like “single-minded pursuit of a top-level goal” is generally important if we have a chance of building AI systems which are sensitive and don’t compete with people.
I mean I also imagine that the agents which survive the best are the ones that are trying to survive. I don’t understand why we’d expect agents that are trying to survive and also accomplish some separate arbitrary infinite-horizon goal would outperform those that are just trying to maintain the conditions necessary for their survival without additional baggage.
To be clear, my position is not “homeostatic agents make good tools and so we should invest efforts in creating them”. My position is “it’s likely that homeostatic agents have significant competitive advantages against unbounded-horizon consequentialist ones, so I expect the future to be full of them, and expect quite a bit of value in figuring out how to make the best of that”.
Ah ok. I was responding to your post’s initial prompt: “I still don’t really intuitively grok why I should expect agents to become better approximated by “single-minded pursuit of a top-level goal” as they gain more capabilities.” (The reason to expect this is that “single-minded pursuit of a top-level goal,” if that goal is survival, could afford evolutionary advantages.)
But I agree entirely that it’d be valuable for us to invest in creating homeostatic agents. Further, I think calling into doubt western/capitalist/individualist notions like “single-minded pursuit of a top-level goal” is generally important if we have a chance of building AI systems which are sensitive and don’t compete with people.