I think “understanding” in humans is an active process that demands cognitive skills we develop with continuous learning. I think you’re right that LLMs are missing “the big picture” and organizing their local concepts to be consistent with it. I don’t think humans do this automatically (per Dweomite’s comment on this post), but that we need to learn skills to do it. I think this a lot of what LLMs are missing (TsviBT’s “dark matter of intelligence”).
I wrote about this in Sapience, understanding, and “AGI” but I wasn’t satisfied and it’s out of date. This is an attempt to do a better and briefer explanation, as a sort of run-up to doing an updated post.
We’ve learned skills for thought management/metacognition/executive function. They’re habits, not beliefs (episodic memories or declarative knowledge), so they’re not obvious to us. We develop “understanding” by using those skills to metaphorically turn over concepts in our minds. This is actively comparing them to memories of data, and other beliefs. Doing this checks their consistency with other things we know. Learning from these investigations improves our future understanding of that concept, and our skills for understanding others.
What LLMs are missing relative to humans is profound right now, but may be all too easy to add adequately to get takeover-capable AGI. Among other things (below), they’re missing cognitive skills that aren’t well-described in the text training set, but may be pretty easy to learn with a system 2 type approach that can be “habitized” with continuous learning. This might be as easy as a little fine-tuning, if the interference problem is adequately solved—and what’s adequate might not be a high bar. Fine-tuning already adds this type of skills, but it seems to produce too much interference for it to keep going. And I don’t know of a full self-teaching loop, although there is constant progress on most or all of the components to build one.
There may be other routes to filling in that missing executive function and active processing for human-like understanding.
This is why I’m terrified of short timelines while most people have slightly longer timelines at this point.
I’ve been thinking about this a lot in light of the excellent critiques of LLM thinking over the last year. My background is “computational cognitive neuroscience,” so comparing LLMs to humans is my main tool for alignment thinking.
When I was just getting acquainted with LLMs in early 2023, my answers were that they’re missing episodic memory (for “snapshot” continuous learning) and “executive function”, a vague term that I’m now thinking is mostly skills for managing cognition. I wrote about this in Capabilities and alignment of LLM cognitive architectures in early 2023. If you can overlook my focus on scaffolding, I think it stands up as a partial analysis of what LLMs are missing and the emergent/ synergistic/ multiplicative advantages of adding those things.
But it’s incomplete. I didn’t emphasize continuous skill learning there, but I now think it’s pretty crucial for how humans develop executive function and therefore understanding. I don’t see a better way to give it to agentic LLMs. RL on tasks could do it, but that has a data problem if it’s not self-directed like human learning is. But there might be other solutions.
I think this is important to figure out. It’s pretty crucial for both timelines and alignment strategy.
I think “understanding” in humans is an active process that demands cognitive skills we develop with continuous learning. I think you’re right that LLMs are missing “the big picture” and organizing their local concepts to be consistent with it. I don’t think humans do this automatically (per Dweomite’s comment on this post), but that we need to learn skills to do it. I think this a lot of what LLMs are missing (TsviBT’s “dark matter of intelligence”).
I wrote about this in Sapience, understanding, and “AGI” but I wasn’t satisfied and it’s out of date. This is an attempt to do a better and briefer explanation, as a sort of run-up to doing an updated post.
We’ve learned skills for thought management/metacognition/executive function. They’re habits, not beliefs (episodic memories or declarative knowledge), so they’re not obvious to us. We develop “understanding” by using those skills to metaphorically turn over concepts in our minds. This is actively comparing them to memories of data, and other beliefs. Doing this checks their consistency with other things we know. Learning from these investigations improves our future understanding of that concept, and our skills for understanding others.
What LLMs are missing relative to humans is profound right now, but may be all too easy to add adequately to get takeover-capable AGI. Among other things (below), they’re missing cognitive skills that aren’t well-described in the text training set, but may be pretty easy to learn with a system 2 type approach that can be “habitized” with continuous learning. This might be as easy as a little fine-tuning, if the interference problem is adequately solved—and what’s adequate might not be a high bar. Fine-tuning already adds this type of skills, but it seems to produce too much interference for it to keep going. And I don’t know of a full self-teaching loop, although there is constant progress on most or all of the components to build one.
There may be other routes to filling in that missing executive function and active processing for human-like understanding.
This is why I’m terrified of short timelines while most people have slightly longer timelines at this point.
I’ve been thinking about this a lot in light of the excellent critiques of LLM thinking over the last year. My background is “computational cognitive neuroscience,” so comparing LLMs to humans is my main tool for alignment thinking.
When I was just getting acquainted with LLMs in early 2023, my answers were that they’re missing episodic memory (for “snapshot” continuous learning) and “executive function”, a vague term that I’m now thinking is mostly skills for managing cognition. I wrote about this in Capabilities and alignment of LLM cognitive architectures in early 2023. If you can overlook my focus on scaffolding, I think it stands up as a partial analysis of what LLMs are missing and the emergent/ synergistic/ multiplicative advantages of adding those things.
But it’s incomplete. I didn’t emphasize continuous skill learning there, but I now think it’s pretty crucial for how humans develop executive function and therefore understanding. I don’t see a better way to give it to agentic LLMs. RL on tasks could do it, but that has a data problem if it’s not self-directed like human learning is. But there might be other solutions.
I think this is important to figure out. It’s pretty crucial for both timelines and alignment strategy.