This seems to make a jump from “the prompt requires agency to execute well” to “the AI develops the cognitive capability for agency”?
In my scenario the AI already has the cognitive capability for agency. It’s just that the capability is latent until the right prompt causes it to be expressed. We’ve seen early examples of this with ChatGPT, where, if you ask it to plan something or think about adversarial scenarios, it will demonstrate agent-ish behavior.
My point is that while current AIs are probably incapable of having agency, future AIs probably will have that capability. Furthermore, we may not be able to tell the difference between an AI that is capable of building a world-model and engaging in long-term goal directed behavior and the current AI systems that mostly aren’t.
In my scenario the AI already has the cognitive capability for agency. It’s just that the capability is latent until the right prompt causes it to be expressed. We’ve seen early examples of this with ChatGPT, where, if you ask it to plan something or think about adversarial scenarios, it will demonstrate agent-ish behavior.
My point is that while current AIs are probably incapable of having agency, future AIs probably will have that capability. Furthermore, we may not be able to tell the difference between an AI that is capable of building a world-model and engaging in long-term goal directed behavior and the current AI systems that mostly aren’t.