This makes sense. I think the important part is not the emergence of agency, but that agency is a convergent route to long-term planning. I’m less worried about intention emerging, and more worried about it being built in to improve capabilities like long-term planning through goal directed search. Agency is also key to how humans do self-directed learning, another super useful ability in just about any domain. I just wrote a short post on the usefulness of agency for self-directed learning: Sapience, understanding, and “AGI”
This makes sense. I think the important part is not the emergence of agency, but that agency is a convergent route to long-term planning. I’m less worried about intention emerging, and more worried about it being built in to improve capabilities like long-term planning through goal directed search. Agency is also key to how humans do self-directed learning, another super useful ability in just about any domain. I just wrote a short post on the usefulness of agency for self-directed learning: Sapience, understanding, and “AGI”