I think you’re pointing to more layers of complexity in how goals will arise in LLM agents.
As for what it all means WRT metacognition that can stabilize the goal structure: I don’t know, but I’ve got some thoughts! They’ll be in the form of a long post I’ve almost finished editing; I plan to publish tomorrow.
Those sources of goals are going to interact in complex ways both during training, as you note, and during chain of thought. No goals are truly arising solely from the chain of thought, since that’s entirely based on the semantics it’s learned from training.
I think you’re pointing to more layers of complexity in how goals will arise in LLM agents.
As for what it all means WRT metacognition that can stabilize the goal structure: I don’t know, but I’ve got some thoughts! They’ll be in the form of a long post I’ve almost finished editing; I plan to publish tomorrow.
Those sources of goals are going to interact in complex ways both during training, as you note, and during chain of thought. No goals are truly arising solely from the chain of thought, since that’s entirely based on the semantics it’s learned from training.