Fourth, and most importantly, if superposition happens more in narrower layers, and if superposition is a cause of adversarial vulnerabilities, this would predict that deep, narrow networks would be less adversarially robust than shallow, wide networks that achieve the same performance and have the same number of parameters. However, Huang et al., (2022) found the exact opposite to be the case.
I’m not sure why the superposition hypothesis would predict that narrower, deeper networks would have more superposition than wider, shallower networks. I don’t think I’ve seen this claim anywhere—if they learn all the same features and have the same number of neurons, I’d expect them to have similar amounts of superposition. Also, can you explain how the feature hypothesis “explains the results from Huang et al.”?
More generally, I think superposition existing in toy models provides a plausible rational for adversarial examples both being very common (even as we scale up models) and also being bugs. Given this and the Elhage et al. (2022) work (which is bayesian evidence towards the bug hypothesis, despite the plausibility of confounders), I’m very surprised you come out with “Verdict: Moderate evidence in favor of the feature hypothesis.”
We talked about this over DMs, but I’ll post a quick reply for the rest of the world. Thanks for the comment.
A lot of how this is interpreted depends on what the exact definition of superposition that one uses and whether it applies to entire networks or single layers. But a key thing I want to highlight is that if a layer represents a certain set amount of information about an example, then they layer must have more information per neuron if it’s thin than if it’s wide. And that is the point I think that the Huang paper helps to make. The fact that deep and thin networks tend to be more robust suggests that representing information more densely w.r.t. neurons in a layer does not make these networks less robust than wide shallow nets.
I’m not sure why the superposition hypothesis would predict that narrower, deeper networks would have more superposition than wider, shallower networks. I don’t think I’ve seen this claim anywhere—if they learn all the same features and have the same number of neurons, I’d expect them to have similar amounts of superposition. Also, can you explain how the feature hypothesis “explains the results from Huang et al.”?
More generally, I think superposition existing in toy models provides a plausible rational for adversarial examples both being very common (even as we scale up models) and also being bugs. Given this and the Elhage et al. (2022) work (which is bayesian evidence towards the bug hypothesis, despite the plausibility of confounders), I’m very surprised you come out with “Verdict: Moderate evidence in favor of the feature hypothesis.”
We talked about this over DMs, but I’ll post a quick reply for the rest of the world. Thanks for the comment.
A lot of how this is interpreted depends on what the exact definition of superposition that one uses and whether it applies to entire networks or single layers. But a key thing I want to highlight is that if a layer represents a certain set amount of information about an example, then they layer must have more information per neuron if it’s thin than if it’s wide. And that is the point I think that the Huang paper helps to make. The fact that deep and thin networks tend to be more robust suggests that representing information more densely w.r.t. neurons in a layer does not make these networks less robust than wide shallow nets.