I’ve been thinking about out of context reasoning lately and wanted to check if my thinking is in line with yours.
For the cases with induction and SFT, the way I’ve been thinking about it is that the model is somehow updating latent structure from its internal world model. So for example, we can conceptualize something like the function f(x) = x // 4 as a latent parent node, and then the observed (x, y) pairs consistent with it as child observations. You train on the (x, y) pairs, and the model infers the latent parent/rule that explains them. I actually thought I remembered seeing a DAG like this in one of your papers or slides, but I can’t find it now, so maybe I hallucinated it.
This can be thought of like a kind of amortization of the inference over that DAG into the weights. Thinking of OOCR in this way makes me think it’s very important, since it seems to point at internal world models in a pretty concrete way.
But this seems different in kind to multi-hop reasoning, which until I read this post I had not thought about as being OOCR. In that case, it feels more like the model is chaining together already-known facts in the forward pass (importantly, with the intermediate step not written into the context). Is there also some non-tortuous way to think about that phenomenon as inference of a latent like in the induction case? Or do you think these are actually pretty different types of things, but still, both are OOCR?
Thanks for writing this up Owain!
I’ve been thinking about out of context reasoning lately and wanted to check if my thinking is in line with yours.
For the cases with induction and SFT, the way I’ve been thinking about it is that the model is somehow updating latent structure from its internal world model. So for example, we can conceptualize something like the function
f(x) = x // 4as a latent parent node, and then the observed(x, y)pairs consistent with it as child observations. You train on the(x, y)pairs, and the model infers the latent parent/rule that explains them. I actually thought I remembered seeing a DAG like this in one of your papers or slides, but I can’t find it now, so maybe I hallucinated it.This can be thought of like a kind of amortization of the inference over that DAG into the weights. Thinking of OOCR in this way makes me think it’s very important, since it seems to point at internal world models in a pretty concrete way.
But this seems different in kind to multi-hop reasoning, which until I read this post I had not thought about as being OOCR. In that case, it feels more like the model is chaining together already-known facts in the forward pass (importantly, with the intermediate step not written into the context). Is there also some non-tortuous way to think about that phenomenon as inference of a latent like in the induction case? Or do you think these are actually pretty different types of things, but still, both are OOCR?