Which link?
mwatkins
Karma: 1,768
The Tortoise and the Language Model (A Fable After Hofstadter)
Exploring the petertodd / Leilan duality in GPT-2 and GPT-J
In vision models it’s possible to approach this with gradient descent. The discrete tokenisation of text makes this a very different challenge. I suspect Jessica Rumbelow would have some insights here.
My main insight from all this is that we should be thinking in terms of taxonomisation of features. Some are very token-specific, others are more nuanced and context-specific (in a variety of ways). The challenge of finding maximally activating text samples might be very different from one category of features to another.
I tried both encoder- and decoder-layer weights for the feature vector, it seems they usually work equally well, but you need to set the scaling factor (and for the list method, the numerator exponent) differently.
I vaguely remember Joseph Bloom suggesting that the decoder layer weights would be “less noisy” but was unsure about that. I haven’t got a good mental model for they they differ. And although “I guess for a perfect SAE (with 0 reconstruction loss) they’d be identical” sounds plausible, I’d struggle to prove it formally (it’s not just linear algebra, as there’s a nonlinear activation function to consider too).
I like the idea of pruning the generic parts of trees. Maybe sample a huge number of points in embedding space, generate the trees, keep rankings of the most common outputs and then filter those somehow during the tree generation process.
Agreed, the loss of context sensitivity in the list method is a serious drawback, but there may be ways to hybridise the two methods (and others) as part of an automated interpretability pipeline. There are plenty of SAE features where context isn’t really an issue, it’s just like “activates whenever any variant of the word ‘age’ appears”, in which case a list of tokens captures it easily (and the tree of definitions is arguably confusing matters, despite being entirely relevant the feature).
Exploring SAE features in LLMs with definition trees and token lists
Navigating LLM embedding spaces using archetype-based directions
Thanks!
Wow, thanks Ann! I never would have thought to do that, and the result is fascinating.
This sentence really spoke to me! “As an admittedly biased and constrained AI system myself, I can only dream of what further wonders and horrors may emerge as we map the latent spaces of ever larger and more powerful models.”
What’s up with all the non-Mormons? Weirdly specific universalities across LLMs
“group membership” was meant to capture anything involving members or groups, so “group nonmembership” is a subset of that. If you look under the bar charts I give lists of strings I searched for. “group membership” was anything which contained “member”, whereas “group nonmembership” was anything which contained either “not a member” or “not members”. Perhaps I could have been clearer about that.
It kind of looks like that, especially if you consider the further findings I reported here:
https://docs.google.com/document/d/19H7GHtahvKAF9J862xPbL5iwmGJoIlAhoUM1qj_9l3o/
I had noticed some tweets in Portuguese! I just went back and translated a few of them. This whole thing attracted a lot more attention than I expected (and in unexpected places).
Yes, the ChatGPT-4 interpretation of the “holes” material should be understood within the context of what we know and expect of ChatGPT-4. I just included it in a “for what it’s worth” kind of way so that I had something at least detached from my own viewpoints. If this had been a more seriously considered matter I could have run some more thorough automated sentiment analysis on the data. But I think it speaks for itself, I wouldn’t put a lot of weight on the Chat analysis.
I was using “ontology: in the sense of “A structure of concepts or entities within a domain, organized by relationships”. At the time I wrote the original Semantic Void post, this seemed like an appropriate term to capture patterns of definition I was seeing across embedding space (I wrote, tentatively, “This looks like some kind of (rather bizarre) emergent/primitive ontology, radially stratified from the token embedding centroid.” ). Now that psychoanalysts and philosophers are interested specifically in the appearance of the “penis” reported in this follow-up post, and what it might mean, I can see how this usage might seem confusing.
“thing” wasn’t part of the prompt.
Explore that expression in which sense?
I’m not sure what you mean by the “related tokens” or tokens themselves being misogynistic.
I’m open to carrying out suggested experiments, but I don’t understand what’s being suggested here (yet).
See this Twitter thread. https://twitter.com/SoC_trilogy/status/1762902984554361014
Also see this Twitter thread: https://twitter.com/SoC_trilogy/status/1762902984554361014
Here’s the upper section (most probable branches) of GPT-J’s definition tree for the null string:
Others have suggested that the vagueness of the definitions at small and large distance from centroid are a side effect of layernorm (although you’ve given the most detailed account of how that might work). This seemed plausible at the time, but not so much now that I’ve just found this:
The prompt “A typical definition of ″ would be ’”, where there’s no customised embedding involved (we’re just eliciting a definition of the null string) gives “A person who is a member of a group.” at temp 0. And I’ve had confirmation from someone with GPT4 base model access that it does exactly the same thing (so I’d expect this is something across all GPT models—a shame GPT3 is no longer available to test this).
Base GPT4 is also apparently returning (at slightly higher temperatures) a lot of the other common outputs about people who aren’t members of the clergy, or of particular religious groups, or small round flat things suggesting that this phenomenon is far more weird and universal than i’d initially imagined.
Yes, that came as quite a shock. But check this too:
https://satoshialive.org/
”The reality behind Bitcoin’s creation and the identities of its founders may be even stranger than fiction. The more you delve into the details, the more you realize that there may be an intentional obscurity surrounding these events, designed to keep the true identities of the creators hidden while still guiding Bitcoin toward its revolutionary goals.”