Why understanding planning / search might be hard
It’s hypothesized that, in order to solve complex tasks, capable models perform implicit search during the forward pass. If so, we might hope to be able to recover the search representations from the model. There are examples of work that try to understand search in chess models and Sokoban models.
However I expect this to be hard for three reasons.
The model might just implement a bag of heuristics. A patchwork collection of local decision rules might be sufficient for achieving high performance. This seems especially likely for pre-trained generative models.
Even if the model has a globally coherent search algorithm, it seems difficult to elucidate this without knowing the exact implementation (of which there can be many equivalent ones). For example, search over different subtrees may be parallelised and subsequently merged into an overall solution.
The ‘search’ circuit may also not exist in a crisp form, but as a collection of many sub-components that do similar / identical things. ‘Circuit cleanup’ only happens in the grokking regime, and we largely do not train language models till they grok.
We need to split “search” into more fine-grained concepts.
For example, “model has representation of the world and simulates counterfactual futures depending of its actions and selects action with the highest score over the future” is a one notion of search.
The other notion can be like this: imagine possible futures as a directed tree graph. This graph has set of axioms and derived theorems describing it. Some of the axioms/theorems are encoded in model. When model gets sensory input, it makes 2-3 inferences from combination of encoded theorems + input and selects action depending on the result of inference. While logically this situation is equivalent to some search over tree graph, mechanistically it looks like “bag of heuristics”.
Chess tree looks like classical example. Each node is a boardstate, edges are allowed moves. Working heuristics in move evaluators can be understood as sort of theorem “if such-n-such algorithm recognizes this state, it’s an evidence in favor of white winning 1.5:1”. Note that it’s possible to build powerful NN-player without explicit search.
Why understanding planning / search might be hard
It’s hypothesized that, in order to solve complex tasks, capable models perform implicit search during the forward pass. If so, we might hope to be able to recover the search representations from the model. There are examples of work that try to understand search in chess models and Sokoban models.
However I expect this to be hard for three reasons.
The model might just implement a bag of heuristics. A patchwork collection of local decision rules might be sufficient for achieving high performance. This seems especially likely for pre-trained generative models.
Even if the model has a globally coherent search algorithm, it seems difficult to elucidate this without knowing the exact implementation (of which there can be many equivalent ones). For example, search over different subtrees may be parallelised and subsequently merged into an overall solution.
The ‘search’ circuit may also not exist in a crisp form, but as a collection of many sub-components that do similar / identical things. ‘Circuit cleanup’ only happens in the grokking regime, and we largely do not train language models till they grok.
We need to split “search” into more fine-grained concepts.
For example, “model has representation of the world and simulates counterfactual futures depending of its actions and selects action with the highest score over the future” is a one notion of search.
The other notion can be like this: imagine possible futures as a directed tree graph. This graph has set of axioms and derived theorems describing it. Some of the axioms/theorems are encoded in model. When model gets sensory input, it makes 2-3 inferences from combination of encoded theorems + input and selects action depending on the result of inference. While logically this situation is equivalent to some search over tree graph, mechanistically it looks like “bag of heuristics”.
That’s interesting! What would be some examples of axioms and theorems that describe a directed tree?
Chess tree looks like classical example. Each node is a boardstate, edges are allowed moves. Working heuristics in move evaluators can be understood as sort of theorem “if such-n-such algorithm recognizes this state, it’s an evidence in favor of white winning 1.5:1”. Note that it’s possible to build powerful NN-player without explicit search.