Goodfire’s new dictionary learning architecture which captures manifolds and seems like the first one to work on toy models of manifold discovery. Strangely enough, I haven’t seen much discussion of it online, and the release wasn’t publicized. Perhaps an extension to LLMs will be released soon?
Thanks for sharing this, very cool to see active research in this direction! It’s exciting to see the field starting to move this way because there is so much potential here.
Getting the geometry right isn’t an easy jump to land, but it’s clear that they’re on to something given their steering successes. The transition of the atomic unit from direction to its own subspace is a rock solid way of handling the linear/nonlinear complexities of navigating manifold routing.
I’m fully in agreement with the authors that SAEs leave so much on the table by ignoring features’ internal geometry. That said, I think they don’t move far enough as they’re still using lossy tools reading concepts off of fit. (To their credit they are doing a very clean implementation, but a clean implementation of a lossy method is still lossy.)
The logical upgrade here is a method that allows for reading the exact manifold off the weights instead of using a dictionary to rank approximations. That’s where things really start to unlock from what I’m seeing.
Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds
Goodfire’s new dictionary learning architecture which captures manifolds and seems like the first one to work on toy models of manifold discovery. Strangely enough, I haven’t seen much discussion of it online, and the release wasn’t publicized. Perhaps an extension to LLMs will be released soon?
Thanks for sharing this, very cool to see active research in this direction! It’s exciting to see the field starting to move this way because there is so much potential here.
Getting the geometry right isn’t an easy jump to land, but it’s clear that they’re on to something given their steering successes. The transition of the atomic unit from direction to its own subspace is a rock solid way of handling the linear/nonlinear complexities of navigating manifold routing.
I’m fully in agreement with the authors that SAEs leave so much on the table by ignoring features’ internal geometry. That said, I think they don’t move far enough as they’re still using lossy tools reading concepts off of fit. (To their credit they are doing a very clean implementation, but a clean implementation of a lossy method is still lossy.)
The logical upgrade here is a method that allows for reading the exact manifold off the weights instead of using a dictionary to rank approximations. That’s where things really start to unlock from what I’m seeing.