Hi Neel! Thanks so much for all these online resources. I’ve been finding them really interesting and helpful.
I have a question about research methods. “How far can you get with really deeply reverse engineering a neuron in a 1 layer (1L) model? (solu-1l, solu-1l-pile or gelu-1l in TransformerLens).”
I’ve loaded up solu-1l in my Jupyter notebook but now feeling a bit lost. For your IOI tutorial, there was a very specific benchmark and error signal. However, when I’m just playing around with a model without a clear capability in mind, it’s harder to know how to measure performance. I could make a list of capabilities/benchmarks, systematically run the model on them, and then pick a capability and start ablating the model and seeing effect on performance. However, I’m then restricted to these predefined capabilities. Like, I’m not even sure what the capabilities of solu-1l are.
I could start feeding solu-1l with random inputs and just “looking” at the attention patterns. But I’m wondering if there’s a more efficient way to do this—or another strategy where research does feel like play, as you describe in your notebook.
Great question! My concrete suggestion is to look for interesting neurons in Neuroscope, as I discuss more in the final post. This is a website I made that shows the text that most activates each neuron in the model (for a ton of open source models), and by looking for interesting neurons, you can hopefully find some hook—find a specific task the model can consistently-ish do, analogous to IOI (with a predictable structure you can generate prompts for, ideally with a somewhat algorithmic flavour—something you could write code to solve). And then do the kinds of things in the IOI notebook. Though note that for a 1L model, you can actually mechanistically look at the weights and break down what the model is doing!
On a meta level, the strategy you want to follow in a situation like this is what I call maximising surface area. You want to explore things and try to get exposed to as many random details about the model behaviour as you can. So that you can then serendipitiously notice something interesting and dig into it. The meta-lesson is that when you feel stuck and meandering, you want to pick some purpose to strive for, but that purpose can just be “put yourself in a situation where you have so much data and context that you can spontaneously stumble across something interesting, and cast a really wide net”. Concretely, you want to look for some kind of task/capability that the model is capable of, so you can then try to reverse-engineer it. And a good way to do this is just to run the model on a bunch of dataset examples and look at what it’s good at, and see if you can find any consistent patterns to dig into. To better explore this, I made a tool to visualise the top 10 tokens predicted for each token in the text in Alan Cooney’s CircuitsVis library. You can filter for interesting text by eg looking for tokens where the model’s log prob for the correct next token is significantly higher than attn-only-1l, to cut things down to where the MLPs matter (I’d cut off the log prob at −6 though, so you don’t just notice when attn-only-1l is really incorrect lol).
Hi Neel! Thanks so much for all these online resources. I’ve been finding them really interesting and helpful.
I have a question about research methods. “How far can you get with really deeply reverse engineering a neuron in a 1 layer (1L) model? (solu-1l, solu-1l-pile or gelu-1l in TransformerLens).”
I’ve loaded up solu-1l in my Jupyter notebook but now feeling a bit lost. For your IOI tutorial, there was a very specific benchmark and error signal. However, when I’m just playing around with a model without a clear capability in mind, it’s harder to know how to measure performance. I could make a list of capabilities/benchmarks, systematically run the model on them, and then pick a capability and start ablating the model and seeing effect on performance. However, I’m then restricted to these predefined capabilities. Like, I’m not even sure what the capabilities of solu-1l are.
I could start feeding solu-1l with random inputs and just “looking” at the attention patterns. But I’m wondering if there’s a more efficient way to do this—or another strategy where research does feel like play, as you describe in your notebook.
Thank you!
Great question! My concrete suggestion is to look for interesting neurons in Neuroscope, as I discuss more in the final post. This is a website I made that shows the text that most activates each neuron in the model (for a ton of open source models), and by looking for interesting neurons, you can hopefully find some hook—find a specific task the model can consistently-ish do, analogous to IOI (with a predictable structure you can generate prompts for, ideally with a somewhat algorithmic flavour—something you could write code to solve). And then do the kinds of things in the IOI notebook. Though note that for a 1L model, you can actually mechanistically look at the weights and break down what the model is doing!
On a meta level, the strategy you want to follow in a situation like this is what I call maximising surface area. You want to explore things and try to get exposed to as many random details about the model behaviour as you can. So that you can then serendipitiously notice something interesting and dig into it. The meta-lesson is that when you feel stuck and meandering, you want to pick some purpose to strive for, but that purpose can just be “put yourself in a situation where you have so much data and context that you can spontaneously stumble across something interesting, and cast a really wide net”. Concretely, you want to look for some kind of task/capability that the model is capable of, so you can then try to reverse-engineer it. And a good way to do this is just to run the model on a bunch of dataset examples and look at what it’s good at, and see if you can find any consistent patterns to dig into. To better explore this, I made a tool to visualise the top 10 tokens predicted for each token in the text in Alan Cooney’s CircuitsVis library. You can filter for interesting text by eg looking for tokens where the model’s log prob for the correct next token is significantly higher than attn-only-1l, to cut things down to where the MLPs matter (I’d cut off the log prob at −6 though, so you don’t just notice when attn-only-1l is really incorrect lol).
Thank you for this response!