I tell an LLM my favorite color. As long as that information is in its context window, it has access to it. As soon as that context rolls off or goes away, the LLM no longer has access to that information.
I build an agent with scaffolding that has a database. I tell it my favorite color. The agent records it in the database. The weights of the LLM are still fixed, but during its base training it learned how to access information. So if I ask it at any point in the future what my favorite color is, it knows. It access the information in the database.
Do you consider this continual learning? If not, why not?
So according to you, a system that could acquire new facts, record them, access them, and use them, continuously in this way would not constitute ‘real’ continuous learning. It could conceivably fill its database with the actionable knowledge of 1000 yet unwritten textbooks, but that wouldn’t be ‘real’ to you.
“wholly new ways of conceptualizing and navigating the world, not just keeping track of what’s going on” are learnable and storable in the way I describe.
How is this type of learning not open-ended? What is limiting it?
Your third criteria seems to be related to unsupervised learning, specifically self-play. Not sure why you’d limit continual learning in this way, either.
You seem to be putting somewhat arbitrary constraints on what constitutes continual learning. Generally, if the system’s knowledge base is fixed, it’s incapable of continuing to learn. If it has the capacity to acquire new knowledge and skills, by whatever means, it continues to learn. You’re narrowing that general idea without really justifying why.
As an analogy, take an adult from 30000 BC, call him Grog, and give him access to a database of “actionable knowledge of 1000 textbooks”, and then tell him to go invent a less expensive solid-state LIDAR system. Will he immediately start making progress? I say “obviously not”.
What would the “actionable knowledge” look like? Maybe one piece of “actionable knowledge” is some fact from the ANSI Z136.1 laser eye safety manual (“For pulsed lasers of 1ns–50μs pulse duration and beam diameter 1 cm, at viewing distance 20 cm, the diffusely reflected beam energy cannot safely exceed 0.022 × CA joules, where CA is the correction factor for IR-A light based on reduced absorption properties of melanin”.) OK, Grog looks at that and immediately has some questions. What does “laser” mean? What is a “pulsed laser”? What does “ns” mean? What does “beam diameter” mean? What does “diffusely reflected” mean? Etc. etc.
This “knowledge” is not in fact “actionable” because Grog can’t make heads or tails of it.
And ditto for pretty much every other item in the database. Right?
What Grog would need to do is spend years developing a deep understanding of optics and lasers and so on before he could even start inventing a new LIDAR system. Of course, that’s what modern LIDAR inventors do: spend years developing understanding. Once Grog has that understanding, then yeah sure, convenient database access to relevant facts would be helpful, just as modern LIDAR inventors do in fact keep the ANSI Z136.1 manual in arm’s reach.
Thus, there’s more to knowledge than lists of facts. It’s ways that the facts all connect to each other in an interconnected web, and it’s ways to think about things, etc.
I claim that this all transfers quite well to LLMs. It’s just that LLMs already have decent “understanding” of everything that humans have ever written down anywhere on the internet or in any book, thanks to pretraining. So in our everyday interactions with LLMs, we don’t as often come across situations where the LLM is flailing around like poor Grog. But see 1, 2.
Sorry, I’m afraid I don’t understand what your analogy is supposed to map to. What is Grog in the context of our conversation? You seem to admit at the end that LLMs are not really at all like Grog, in that Grog has no underlying bedrock of understanding, while modern LLMs do.
Thus, there’s more to knowledge than lists of facts. It’s ways that the facts all connect to each other in an interconnected web, and it’s ways to think about things, etc.
I’ll agree with this definition. If you’ll agree that knowledge can exist in written form and textbooks often embody exactly what you describe. They are very rarely ‘lists of facts’. More often than not, they are logically curated, organized explanations of phenomenon and events, along with rich descriptions of their connections and interactions. You seem to be preferentially upselling knowledge that is stored in synaptic weights while drastically downplaying knowledge recorded in other mediums. Why?
What is Grog in the context of our conversation? You seem to admit at the end that LLMs are not really at all like Grog, in that Grog has no underlying bedrock of understanding, while modern LLMs do.
Grog understands some things (e.g. intuitive physics) but not others (e.g. pulsed lasers). Likewise, LLMs understand some things (e.g. pulsed lasers) but not others (e.g. some new field of science that hasn’t been invented yet). Right? We’re not at the end of history, where everything that can possibly be understood is already understood, and there’s nothing left.
If I hibernated you until the year 2100, and then woke you up and gave you a database with “actionable knowledge” from 1000 textbooks of [yet-to-be-invented fields of science], and asked you to engineer a state-of-the-art [device that no one today has even conceived of], then you would be just as helpless as Grog. You would have to learn the new fields until you understood them, which might take years, before you could even start on the task. This process involves changing the “weights” in your brain. I.e., you would need “real” learning. The database is not a replacement for that.
So think of it this way: there’s some set of things that are understood (by anyone), and that set of things is not increased via a system for pulling up facts from a database. Otherwise Grog would be able to immediately design LIDAR. And yet, humans are able to increase the set of things that are understood, over time. After all, “the set of things that are understood” sure is bigger today than it was 1000 years ago, and will be bigger still in 2100. So evidently humans are doing something very important that is entirely different from what can be done with database systems. And that thing is what I’m calling “real” continual learning.
Its about the homogeneity of the data representation. An argument you could make is, if the hippocampus “stores” data in the neocortex, isn’t the neocortex “just a database”? Even the brain uses different forms of neural networks. At that point the only distinction is that those neural networks share the same protocol of communication (spiking, dendrites, synapses, etc) whereas an llm breaks the protocol when it switches to tool use.
Another distinction, however, is that in a continual learning system, the new data affects the previous set, in a way that “code is data and data is code”, while an llm accessing a database doesn’t affect the capabilities already learnt by the llm. But what if the llm accesses e.g. a prompt persona from the database? At that point the important question becomes: is ICL really “learning”? I would say not, it is just a preference optimization over what the llm has already learned.
An attempt to reach a more formal definition of continual learning could be that ultimately it is a system that is irreversibly updated (it has no inverse) from its learning process.
I tell an LLM my favorite color. As long as that information is in its context window, it has access to it. As soon as that context rolls off or goes away, the LLM no longer has access to that information.
I build an agent with scaffolding that has a database. I tell it my favorite color. The agent records it in the database. The weights of the LLM are still fixed, but during its base training it learned how to access information. So if I ask it at any point in the future what my favorite color is, it knows. It access the information in the database.
Do you consider this continual learning? If not, why not?
See everything I wrote in the section “Some intuitions on how to think about ‘real’ continual learning”. The thing you’re describing is definitely not (what I’m calling) “real” continual learning.
Should the thing you’re describing be called “continual learning” at all? No opinion. Call it whatever you want.
So according to you, a system that could acquire new facts, record them, access them, and use them, continuously in this way would not constitute ‘real’ continuous learning. It could conceivably fill its database with the actionable knowledge of 1000 yet unwritten textbooks, but that wouldn’t be ‘real’ to you.
“wholly new ways of conceptualizing and navigating the world, not just keeping track of what’s going on” are learnable and storable in the way I describe.
How is this type of learning not open-ended? What is limiting it?
Your third criteria seems to be related to unsupervised learning, specifically self-play. Not sure why you’d limit continual learning in this way, either.
You seem to be putting somewhat arbitrary constraints on what constitutes continual learning. Generally, if the system’s knowledge base is fixed, it’s incapable of continuing to learn. If it has the capacity to acquire new knowledge and skills, by whatever means, it continues to learn. You’re narrowing that general idea without really justifying why.
As an analogy, take an adult from 30000 BC, call him Grog, and give him access to a database of “actionable knowledge of 1000 textbooks”, and then tell him to go invent a less expensive solid-state LIDAR system. Will he immediately start making progress? I say “obviously not”.
What would the “actionable knowledge” look like? Maybe one piece of “actionable knowledge” is some fact from the ANSI Z136.1 laser eye safety manual (“For pulsed lasers of 1ns–50μs pulse duration and beam diameter 1 cm, at viewing distance 20 cm, the diffusely reflected beam energy cannot safely exceed 0.022 × CA joules, where CA is the correction factor for IR-A light based on reduced absorption properties of melanin”.) OK, Grog looks at that and immediately has some questions. What does “laser” mean? What is a “pulsed laser”? What does “ns” mean? What does “beam diameter” mean? What does “diffusely reflected” mean? Etc. etc.
This “knowledge” is not in fact “actionable” because Grog can’t make heads or tails of it.
And ditto for pretty much every other item in the database. Right?
What Grog would need to do is spend years developing a deep understanding of optics and lasers and so on before he could even start inventing a new LIDAR system. Of course, that’s what modern LIDAR inventors do: spend years developing understanding. Once Grog has that understanding, then yeah sure, convenient database access to relevant facts would be helpful, just as modern LIDAR inventors do in fact keep the ANSI Z136.1 manual in arm’s reach.
Thus, there’s more to knowledge than lists of facts. It’s ways that the facts all connect to each other in an interconnected web, and it’s ways to think about things, etc.
I claim that this all transfers quite well to LLMs. It’s just that LLMs already have decent “understanding” of everything that humans have ever written down anywhere on the internet or in any book, thanks to pretraining. So in our everyday interactions with LLMs, we don’t as often come across situations where the LLM is flailing around like poor Grog. But see 1, 2.
Sorry, I’m afraid I don’t understand what your analogy is supposed to map to. What is Grog in the context of our conversation? You seem to admit at the end that LLMs are not really at all like Grog, in that Grog has no underlying bedrock of understanding, while modern LLMs do.
I’ll agree with this definition. If you’ll agree that knowledge can exist in written form and textbooks often embody exactly what you describe. They are very rarely ‘lists of facts’. More often than not, they are logically curated, organized explanations of phenomenon and events, along with rich descriptions of their connections and interactions. You seem to be preferentially upselling knowledge that is stored in synaptic weights while drastically downplaying knowledge recorded in other mediums. Why?
Grog understands some things (e.g. intuitive physics) but not others (e.g. pulsed lasers). Likewise, LLMs understand some things (e.g. pulsed lasers) but not others (e.g. some new field of science that hasn’t been invented yet). Right? We’re not at the end of history, where everything that can possibly be understood is already understood, and there’s nothing left.
If I hibernated you until the year 2100, and then woke you up and gave you a database with “actionable knowledge” from 1000 textbooks of [yet-to-be-invented fields of science], and asked you to engineer a state-of-the-art [device that no one today has even conceived of], then you would be just as helpless as Grog. You would have to learn the new fields until you understood them, which might take years, before you could even start on the task. This process involves changing the “weights” in your brain. I.e., you would need “real” learning. The database is not a replacement for that.
So think of it this way: there’s some set of things that are understood (by anyone), and that set of things is not increased via a system for pulling up facts from a database. Otherwise Grog would be able to immediately design LIDAR. And yet, humans are able to increase the set of things that are understood, over time. After all, “the set of things that are understood” sure is bigger today than it was 1000 years ago, and will be bigger still in 2100. So evidently humans are doing something very important that is entirely different from what can be done with database systems. And that thing is what I’m calling “real” continual learning.
Its about the homogeneity of the data representation. An argument you could make is, if the hippocampus “stores” data in the neocortex, isn’t the neocortex “just a database”? Even the brain uses different forms of neural networks. At that point the only distinction is that those neural networks share the same protocol of communication (spiking, dendrites, synapses, etc) whereas an llm breaks the protocol when it switches to tool use.
Another distinction, however, is that in a continual learning system, the new data affects the previous set, in a way that “code is data and data is code”, while an llm accessing a database doesn’t affect the capabilities already learnt by the llm. But what if the llm accesses e.g. a prompt persona from the database? At that point the important question becomes: is ICL really “learning”? I would say not, it is just a preference optimization over what the llm has already learned.
An attempt to reach a more formal definition of continual learning could be that ultimately it is a system that is irreversibly updated (it has no inverse) from its learning process.