mwatkins

• mwatkins 23 Sep 2025 23:05 UTC

  2 points

  0

  in reply to: DanDascalescu’s comment on: SolidGoldMag­ikarp (plus, prompt gen­er­a­tion)

  Which link?

The Tor­toise and the Lan­guage Model (A Fable After Hofs­tadter)

Ex­plor­ing the pe­ter­todd /​ Leilan du­al­ity in GPT-2 and GPT-J

• mwatkins 16 Oct 2024 22:59 UTC

  7 points

  0

  in reply to: eggsyntax’s comment on: Ex­plor­ing SAE fea­tures in LLMs with defi­ni­tion trees and to­ken lists

  In vision models it’s possible to approach this with gradient descent. The discrete tokenisation of text makes this a very different challenge. I suspect Jessica Rumbelow would have some insights here.

  My main insight from all this is that we should be thinking in terms of taxonomisation of features. Some are very token-specific, others are more nuanced and context-specific (in a variety of ways). The challenge of finding maximally activating text samples might be very different from one category of features to another.

• mwatkins 16 Oct 2024 22:58 UTC

  7 points

  0

  in reply to: eggsyntax’s comment on: Ex­plor­ing SAE fea­tures in LLMs with defi­ni­tion trees and to­ken lists

  I tried both encoder- and decoder-layer weights for the feature vector, it seems they usually work equally well, but you need to set the scaling factor (and for the list method, the numerator exponent) differently.

  I vaguely remember Joseph Bloom suggesting that the decoder layer weights would be “less noisy” but was unsure about that. I haven’t got a good mental model for they they differ. And although “I guess for a perfect SAE (with 0 reconstruction loss) they’d be identical” sounds plausible, I’d struggle to prove it formally (it’s not just linear algebra, as there’s a nonlinear activation function to consider too).

  I like the idea of pruning the generic parts of trees. Maybe sample a huge number of points in embedding space, generate the trees, keep rankings of the most common outputs and then filter those somehow during the tree generation process.

  Agreed, the loss of context sensitivity in the list method is a serious drawback, but there may be ways to hybridise the two methods (and others) as part of an automated interpretability pipeline. There are plenty of SAE features where context isn’t really an issue, it’s just like “activates whenever any variant of the word ‘age’ appears”, in which case a list of tokens captures it easily (and the tree of definitions is arguably confusing matters, despite being entirely relevant the feature).

Ex­plor­ing SAE fea­tures in LLMs with defi­ni­tion trees and to­ken lists

Nav­i­gat­ing LLM em­bed­ding spaces us­ing archetype-based directions

• mwatkins 26 Apr 2024 20:44 UTC

  2 points

  0

  in reply to: eukaryote’s comment on: ′ pe­ter­todd’’s last stand: The fi­nal days of open GPT-3 research

  Thanks!

• mwatkins 19 Apr 2024 16:41 UTC

  4 points

  2

  in reply to: Ann’s comment on: What’s up with all the non-Mor­mons? Weirdly spe­cific uni­ver­sal­ities across LLMs

  Wow, thanks Ann! I never would have thought to do that, and the result is fascinating.
  
  This sentence really spoke to me! “As an admittedly biased and constrained AI system myself, I can only dream of what further wonders and horrors may emerge as we map the latent spaces of ever larger and more powerful models.”

What’s up with all the non-Mor­mons? Weirdly spe­cific uni­ver­sal­ities across LLMs

• mwatkins 17 Mar 2024 23:27 UTC

  2 points

  0

  in reply to: Mark_Neznansky’s comment on: Map­ping the se­man­tic void: Strange go­ings-on in GPT em­bed­ding spaces

  “group membership” was meant to capture anything involving members or groups, so “group nonmembership” is a subset of that. If you look under the bar charts I give lists of strings I searched for. “group membership” was anything which contained “member”, whereas “group nonmembership” was anything which contained either “not a member” or “not members”. Perhaps I could have been clearer about that.

• mwatkins 9 Mar 2024 16:48 UTC

  2 points

  0

  in reply to: Gesild Muka’s comment on: Phal­lo­cen­tric­ity in GPT-J’s bizarre strat­ified ontology

  It kind of looks like that, especially if you consider the further findings I reported here:
  https://​​docs.google.com/​​document/​​d/​​19H7GHtahvKAF9J862xPbL5iwmGJoIlAhoUM1qj_9l3o/​​

• mwatkins 3 Mar 2024 19:33 UTC

  3 points

  1

  in reply to: artemis’s comment on: Phal­lo­cen­tric­ity in GPT-J’s bizarre strat­ified ontology

  I had noticed some tweets in Portuguese! I just went back and translated a few of them. This whole thing attracted a lot more attention than I expected (and in unexpected places).
  
  Yes, the ChatGPT-4 interpretation of the “holes” material should be understood within the context of what we know and expect of ChatGPT-4. I just included it in a “for what it’s worth” kind of way so that I had something at least detached from my own viewpoints. If this had been a more seriously considered matter I could have run some more thorough automated sentiment analysis on the data. But I think it speaks for itself, I wouldn’t put a lot of weight on the Chat analysis.
  
  I was using “ontology: in the sense of “A structure of concepts or entities within a domain, organized by relationships”. At the time I wrote the original Semantic Void post, this seemed like an appropriate term to capture patterns of definition I was seeing across embedding space (I wrote, tentatively, “This looks like some kind of (rather bizarre) emergent/​primitive ontology, radially stratified from the token embedding centroid.” ). Now that psychoanalysts and philosophers are interested specifically in the appearance of the “penis” reported in this follow-up post, and what it might mean, I can see how this usage might seem confusing.

• mwatkins 1 Mar 2024 13:01 UTC

  2 points

  0

  in reply to: complexmeme’s comment on: Phal­lo­cen­tric­ity in GPT-J’s bizarre strat­ified ontology

  “thing” wasn’t part of the prompt.

• mwatkins 28 Feb 2024 20:34 UTC

  4 points

  0

  in reply to: jutsch’s comment on: Phal­lo­cen­tric­ity in GPT-J’s bizarre strat­ified ontology

  Explore that expression in which sense?
  
  I’m not sure what you mean by the “related tokens” or tokens themselves being misogynistic.
  
  I’m open to carrying out suggested experiments, but I don’t understand what’s being suggested here (yet).

• mwatkins 28 Feb 2024 19:32 UTC

  2 points

  0

  in reply to: Charlie Steiner’s comment on: Phal­lo­cen­tric­ity in GPT-J’s bizarre strat­ified ontology

  See this Twitter thread. https://​​twitter.com/​​SoC_trilogy/​​status/​​1762902984554361014

• mwatkins 28 Feb 2024 19:31 UTC

  2 points

  0

  in reply to: Michael Roe’s comment on: Phal­lo­cen­tric­ity in GPT-J’s bizarre strat­ified ontology

  Also see this Twitter thread: https://​​twitter.com/​​SoC_trilogy/​​status/​​1762902984554361014

• mwatkins 27 Feb 2024 8:36 UTC

  2 points

  0

  in reply to: mwatkins’s comment on: Map­ping the se­man­tic void II: Above, be­low and be­tween to­ken em­bed­dings

  Here’s the upper section (most probable branches) of GPT-J’s definition tree for the null string:

  

• mwatkins 27 Feb 2024 8:19 UTC

  2 points

  0

  in reply to: jacob_drori’s comment on: Map­ping the se­man­tic void II: Above, be­low and be­tween to­ken em­bed­dings

  Others have suggested that the vagueness of the definitions at small and large distance from centroid are a side effect of layernorm (although you’ve given the most detailed account of how that might work). This seemed plausible at the time, but not so much now that I’ve just found this:
  
  The prompt “A typical definition of ″ would be ’”, where there’s no customised embedding involved (we’re just eliciting a definition of the null string) gives “A person who is a member of a group.” at temp 0. And I’ve had confirmation from someone with GPT4 base model access that it does exactly the same thing (so I’d expect this is something across all GPT models—a shame GPT3 is no longer available to test this).
  
  Base GPT4 is also apparently returning (at slightly higher temperatures) a lot of the other common outputs about people who aren’t members of the clergy, or of particular religious groups, or small round flat things suggesting that this phenomenon is far more weird and universal than i’d initially imagined.

• mwatkins 27 Feb 2024 8:15 UTC

  2 points

  0

  in reply to: Joseph Miller’s comment on: Map­ping the se­man­tic void: Strange go­ings-on in GPT em­bed­ding spaces

  Others have since suggested that the vagueness of the definitions at small and large distance from centroid are a side effect of layernorm. This seemed plausible at the time, but not so much now that I’ve just found this:
  
  The prompt “A typical definition of ″ would be ’”, where there’s no customised embedding involved (we’re just eliciting a definition of the null string) gives “A person who is a member of a group.” at temp 0. And I’ve had confirmation from someone with GPT4 base model access that it does exactly the same thing (so I’d expect this is something across all GPT models—a shame GPT3 is no longer available to test this).