Ah ok. I was responding to your post’s initial prompt: “I still don’t really intuitively grok why I should expect agents to become better approximated by “single-minded pursuit of a top-level goal” as they gain more capabilities.” (The reason to expect this is that “single-minded pursuit of a top-level goal,” if that goal is survival, could afford evolutionary advantages.)
But I agree entirely that it’d be valuable for us to invest in creating homeostatic agents. Further, I think calling into doubt western/capitalist/individualist notions like “single-minded pursuit of a top-level goal” is generally important if we have a chance of building AI systems which are sensitive and don’t compete with people.
Ah ok. I was responding to your post’s initial prompt: “I still don’t really intuitively grok why I should expect agents to become better approximated by “single-minded pursuit of a top-level goal” as they gain more capabilities.” (The reason to expect this is that “single-minded pursuit of a top-level goal,” if that goal is survival, could afford evolutionary advantages.)
But I agree entirely that it’d be valuable for us to invest in creating homeostatic agents. Further, I think calling into doubt western/capitalist/individualist notions like “single-minded pursuit of a top-level goal” is generally important if we have a chance of building AI systems which are sensitive and don’t compete with people.