> So, my question is: what might generalization look like from the belief-state perspective?
This is a great question that our Simplex team is actively thinking about from several angles. The main angles I’m thinking about are abstraction and composition, with a practical addendum of subspace superposition. These tricks likely do not exhaust the ways in which intelligences come to generalize, but they are concrete parts of the problem that are probably important contributors. I would welcome other ideas on this that seem similarly foundational!
Among these, I’m currently most energized by a minimal model of abstraction, which we obtain via combining the insights from (i) this post on multiple ergodic components with (ii) those of our “Transformers learn factored representations” paper. We’ll have some work out on this soon, but I’m happy to share the basic idea:
An intelligent agent must be able to make predictions and take actions in an immensely complex world. How can this ever be tractable for an intelligence? It’s useful to recognize that the world is complex in the following way: The agent encounters many different scenarios—i.e., many ergodic components—and each of these scenarios comes to life via the contributions of many co-active parts (i.e., factors). In a relatively simple world, the agent could develop completely new circuitry to deal with each scenario. However, when demanding competency in more complex worlds, the combinatorial explosion of things to track make this “memorizing” (and I’ll argue “redundant”) approach untenable given reasonable memory/capacity constraints.
Crucially, there’s an intelligent hack. Some of the encountered parts will be [approximately] shared across different scenarios. These shared parts are useful abstractions, and can be factored out into a single reusable concept in the agent’s mind.
In our experiments so far, we indeed see that neural networks reliably learn to factor out these abstractions, creating shared subspaces that are reused across scenarios as the combinatorics push against the number of available dimensions in the model’s representation space.
Flexibly composing learned abstractions allows models to generalize somewhat sensibly in novel scenarios. There’s a beautiful geometric description of this all.
As capacity constraints are further stressed, the superposition hypothesis can also be refined in light of these results to account for sparse multi-dimensional subspaces. But we’ll flesh out the other parts of the story first.
> So, my question is: what might generalization look like from the belief-state perspective?
This is a great question that our Simplex team is actively thinking about from several angles. The main angles I’m thinking about are abstraction and composition, with a practical addendum of subspace superposition. These tricks likely do not exhaust the ways in which intelligences come to generalize, but they are concrete parts of the problem that are probably important contributors. I would welcome other ideas on this that seem similarly foundational!
Among these, I’m currently most energized by a minimal model of abstraction, which we obtain via combining the insights from (i) this post on multiple ergodic components with (ii) those of our “Transformers learn factored representations” paper. We’ll have some work out on this soon, but I’m happy to share the basic idea:
An intelligent agent must be able to make predictions and take actions in an immensely complex world. How can this ever be tractable for an intelligence? It’s useful to recognize that the world is complex in the following way: The agent encounters many different scenarios—i.e., many ergodic components—and each of these scenarios comes to life via the contributions of many co-active parts (i.e., factors). In a relatively simple world, the agent could develop completely new circuitry to deal with each scenario. However, when demanding competency in more complex worlds, the combinatorial explosion of things to track make this “memorizing” (and I’ll argue “redundant”) approach untenable given reasonable memory/capacity constraints.
Crucially, there’s an intelligent hack. Some of the encountered parts will be [approximately] shared across different scenarios. These shared parts are useful abstractions, and can be factored out into a single reusable concept in the agent’s mind.
In our experiments so far, we indeed see that neural networks reliably learn to factor out these abstractions, creating shared subspaces that are reused across scenarios as the combinatorics push against the number of available dimensions in the model’s representation space.
Flexibly composing learned abstractions allows models to generalize somewhat sensibly in novel scenarios. There’s a beautiful geometric description of this all.
As capacity constraints are further stressed, the superposition hypothesis can also be refined in light of these results to account for sparse multi-dimensional subspaces. But we’ll flesh out the other parts of the story first.