Out-of-context reasoning, the phenomenon where models can learn much more general, unifying structure when fine-tuned on something fairly specific, was a pretty important update to my mental model of how neural networks work.
This paper wasn’t the first, but it was one of the more clean and compelling early examples (though emergent misalignment is now the most famous).
After staring at it for a while, I now feel less surprised by out-of-context reasoning. Mechanistically, there’s no reason the model couldn’t learn the generalizing solution. And on a task like this, the generalizing solution is just simpler and more efficient, and there’s gradient signal for it (at least if there’s a linear representation), so it’s natural to learn it. But it’s easy to say something’s obvious in hindsight. I would not have predicted this, and it has improved my mental model of neural networks. I think this is important and valuable!
Out-of-context reasoning, the phenomenon where models can learn much more general, unifying structure when fine-tuned on something fairly specific, was a pretty important update to my mental model of how neural networks work.
This paper wasn’t the first, but it was one of the more clean and compelling early examples (though emergent misalignment is now the most famous).
After staring at it for a while, I now feel less surprised by out-of-context reasoning. Mechanistically, there’s no reason the model couldn’t learn the generalizing solution. And on a task like this, the generalizing solution is just simpler and more efficient, and there’s gradient signal for it (at least if there’s a linear representation), so it’s natural to learn it. But it’s easy to say something’s obvious in hindsight. I would not have predicted this, and it has improved my mental model of neural networks. I think this is important and valuable!