Thanks for this! I’ve had similar things on my mind and not had a good way to communicate them to people I’m communicating with. I think this cluster ideas around ‘autonomy’ is pointing at an important point. One which I’m quite glad isn’t being actively explored on the forefront of ML research, actually. I do think that this is a critical piece of AGI, and that deliberate attempts this under-explored topic would probably turn up some low-hanging fruit. I also think it would be bad if we ‘stumble’ onto such an agent without realizing we’ve done so.
I feel like ‘autonomy’ is a decent but not quite right name. Exploring for succinct ways to describe the cluster of ideas, my first thoughts are ‘temporally coherent consequentialism in pursuit of persistent goals’, or ‘long-term goal pursuit across varying tasks with online-learning of meta-strategies’?
Anyway, I think this is exactly what we, as a society, shouldn’t pursue until we’ve got a much better handle on AI alignment.
It might be that the ‘goals’ part of what nostalgebraist is waving at is separable from the temporal coherence part.
I mean this in the sense that GPT-3 doesn’t have goals; obviously you can debate that one way or another, but consider all the attempts to make a long document transformer. The generative models have two kinds of memory, essentially: the weights, analogous to long term memory, and the context window, analogous to working memory. There either needs to be some kind of continuous training/fine-tuning of the weights in production, or it needs a third form of episodic memory where the AI can remember the context (“War with Eurasia ended, we have always been at war with Eastasia”). These improvements could make GPT-3 able to write a coherent novel, plausibly without making it any more agentic.
Thanks for this! I’ve had similar things on my mind and not had a good way to communicate them to people I’m communicating with. I think this cluster ideas around ‘autonomy’ is pointing at an important point. One which I’m quite glad isn’t being actively explored on the forefront of ML research, actually. I do think that this is a critical piece of AGI, and that deliberate attempts this under-explored topic would probably turn up some low-hanging fruit. I also think it would be bad if we ‘stumble’ onto such an agent without realizing we’ve done so.
I feel like ‘autonomy’ is a decent but not quite right name. Exploring for succinct ways to describe the cluster of ideas, my first thoughts are ‘temporally coherent consequentialism in pursuit of persistent goals’, or ‘long-term goal pursuit across varying tasks with online-learning of meta-strategies’?
Anyway, I think this is exactly what we, as a society, shouldn’t pursue until we’ve got a much better handle on AI alignment.
It might be that the ‘goals’ part of what nostalgebraist is waving at is separable from the temporal coherence part.
I mean this in the sense that GPT-3 doesn’t have goals; obviously you can debate that one way or another, but consider all the attempts to make a long document transformer. The generative models have two kinds of memory, essentially: the weights, analogous to long term memory, and the context window, analogous to working memory. There either needs to be some kind of continuous training/fine-tuning of the weights in production, or it needs a third form of episodic memory where the AI can remember the context (“War with Eurasia ended, we have always been at war with Eastasia”). These improvements could make GPT-3 able to write a coherent novel, plausibly without making it any more agentic.