Logan Riggs

• Logan Riggs 23 Jul 2024 18:00 UTC

  LW: 2 AF: 1

  0

  AF

  on: JumpReLU SAEs + Early Ac­cess to Gemma 2 SAEs

  Did y’all do any ablations on your loss terms. For example:
  1. JumpReLU() → ReLU
  2. L0 (w/​ STE) → L1
  
  I’d be curious to see if the pareto improvements and high frequency features are due to one, the other, or both

• Logan Riggs 14 Jul 2024 19:39 UTC

  3 points

  1

  on: An In­tro­duc­tion to Rep­re­sen­ta­tion Eng­ineer­ing—an ac­ti­va­tion-based paradigm for con­trol­ling LLMs

  On activation patching:

  The most salient difference is that RE doesn’t completely change the activations, but merely adds to them. Thus, RE aims to find how a certain concept is represented, while activation patching serves as an ablation of the function of a specific layer or neuron. Thus activation patching doesn’t directly tell you where to find the representation of a concept.

  I’m pretty sure both methods give you some approximate location of the representation. RE is typically done on many layers & then picks the best layer. Activation patching ablates each layer & shows you which one is most important for the counterfactual. I would trust the result of patching more than RE for location of a representation (maybe grounded out as what a SAE would find?) due to being more in-distribution.

• Logan Riggs 14 Jul 2024 19:19 UTC

  8 points

  0

  on: An In­tro­duc­tion to Rep­re­sen­ta­tion Eng­ineer­ing—an ac­ti­va­tion-based paradigm for con­trol­ling LLMs

  Strong upvoted to get from −6 to 0 karma. Would be great if someone who downvotes could explain?

  My read of the paper is this is a topic Jan is researching, and they wrote up their own lit review for their own sake and other’s if they’re interested in the topic which isn’t negative karma-worthy.

• Logan Riggs 5 Jul 2024 18:30 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: gwern’s comment on: In­ter­pret­ing Prefer­ence Models w/​ Sparse Autoencoders

  Regarding urls, I think this is a mix of the HH dataset being non-ideal & the PM not being a great discriminator of chosen vs rejected reward (see nostalgebraist’s comment & my response)

  I do think SAE’s find the relevant features, but inefficiently compressed (see Josh & Isaac’s work on days of the week circle features). So an ideal SAE (or alternative architecture) would not separate these features. Relatedly, many of the features that had high url-relevant reward had above-random cos-sim with each other.

  [I also think the SAE’s could be optimized to trade off some reconstruction loss for reward-difference loss which I expect to show a cleaner effect on the reward]

• Logan Riggs 5 Jul 2024 18:19 UTC

  LW: 4 AF: 2

  0

  AF

  in reply to: nostalgebraist’s comment on: In­ter­pret­ing Prefer­ence Models w/​ Sparse Autoencoders

  The PM is pretty bad (it’s trained on hh).

  

  It’s actually only trained after the first 20k/​156k datapoints in hh, which moves the mean reward-diff from 1.04 → 1.36 if you only calculate over that remaining ~136k subset.

  My understanding is there’s 3 bad things:
  1. the hh dataset is inconsistent
  2. The PM doesn’t separate chosen vs rejected very well (as shown above)
  3. The PM is GPT-J (7B parameter model) which doesn’t have the most complex features to choose from.

  The in-distribution argument is most likely the case for the “Thank you. My pleasure” case, because the assistant never (AFAIK, I didn’t check) said that phrase as a response. Only “My pleasure” after the user said ” thank you”.

• Logan Riggs 5 Jul 2024 17:56 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: Fabien Roger’s comment on: In­ter­pret­ing Prefer­ence Models w/​ Sparse Autoencoders

  I prefer when they are directly mentioned in the post/​paper!

  That would be a more honest picture. The simplest change I could think of was adding it to the high-level takeaways.

  I do think you could use SAE features to beat that baseline if done in the way specified by General Takeaways. Specifically, if you have a completion that seems to do unjustifiably better, then you can find all feature’s effects on the rewards that were different than your baseline completion.

  Features help come up with hypotheses, but also isolates the effect. If do have a specific hypothesis as mentioned, then you should be able to find features that capture that hypothesis (if SAEs are doing their job). When you create some alternative completion based on your hypothesis, you might unknowingly add/​remove additional negative & positive features e.g. just wanting to remove completion-length, you also remove the end-of-sentence punctuation.

  In general, I think it’s hard to come up with the perfect counterfactual, but SAE’s at least let you know if you’re adding or removing specific reward-relevant features in your counterfactual completions.

• Logan Riggs 3 Jul 2024 15:34 UTC

  LW: 8 AF: 5

  0

  AF

  in reply to: Fabien Roger’s comment on: In­ter­pret­ing Prefer­ence Models w/​ Sparse Autoencoders

  Thanks!

  There were some features that didn’t work, specifically ones that activated on movie names & famous people’s names, which I couldn’t get to work. Currently I think they’re actually part of a “items in a list” group of reward-relevant features (like the urls were), but I didn’t attempt to change prompts based off items in a list.

  For “unsupervised find spurious features over a large dataset” my prior is low given my current implementation (ie I didn’t find all the reward-relevant features).

  However, this could be improved with more compute, SAEs over layers, data, and better filtering of the resulting feature results (and better versions of SAEs that e.g. fix feature splitting, train directly for reward).

  From my (quick) read, it’s not obvious how well this approach compares to the baseline of “just look at things the model likes and try to understand the spurious features of the PM”

  From this section, you could augment this with SAE features by finding the features relevant for causing one completion to be different than the other. I think this is the most straightforwardly useful application. A couple of gotcha’s:

  1. Some features are outlier dimensions or high-frequency features which will affect both completions (or even most text), so include some baselines which shouldn’t be affected (which requires a hypothesis)

  2. You should look over multiple layers (though if you do multiple residual stream SAEs you’ll find near-duplicate features)

  What links here?

  • In­ter­pret­ing Prefer­ence Models w/​ Sparse Autoencoders by Logan Riggs (1 Jul 2024 21:35 UTC; 71 points)

• Logan Riggs 2 Jul 2024 15:31 UTC

  2 points

  0

  in reply to: Joseph Bloom’s comment on: In­ter­pret­ing Prefer­ence Models w/​ Sparse Autoencoders

  Fixed! Thanks:)

In­ter­pret­ing Prefer­ence Models w/​ Sparse Autoencoders

• Logan Riggs 20 Jun 2024 22:02 UTC

  9 points

  0

  on: In­ter­pret­ing and Steer­ing Fea­tures in Images

  Really cool work! Some general thoughts from training SAEs on LLMs that might not carry over:

  L0 vs reconstruction

  Your variance explained metric only makes sense for a given sparsity (ie L0). For example, you can get near perfect variance explained if you set your sparsity term to 0, but then the SAE is like an identity function & the features aren’t meaningful.

  In my initial experiments, we swept over various L0s & checked which ones looked most monosemantic (ie had the same meaning across all activations), when sampling 30 features. We found 20-100 being a good L0 for LLMs of d_model=512. I’m curious how this translates to text.

  Dead Features

  I believe you can do Leo Gao’s topk w/​ tied-initialization scheme to (1) tightly control your L0 & (2) have less dead features w/​o doing ghost grads. Gao et al notice that this tied-init (ie setting encoder to decoder transposed initially) led to little dead features for small models, which your d_model of 1k is kind of small.

  Nora has an implementation here (though you’d need to integrate w/​ your vision models)

  Icon Explanation

  I didn’t really understand your icon image… Oh I get it. The template is the far left & the other three are three different features that you clamp to a large value generating from that template. Cool idea! (maybe separate the 3 features from the template or do a 1x3 matrix-table for clarity?)

  Other Interp Ideas

  Feature ablation—Take the top-activating images for a feature, ablate the feature ( by reconstructing w/​o that feature & do the same residual add-in thing you found useful), and see the resulting generation.

  Relevant Inputs—What pixels or something are causally responsible for activating this feature? There’s gotta be some literature on input-causal attribution on the output class in image models, then you’re just applying that to your features instead.

  For instance, the subject of one photo could be transferred to another. We could adjust the time of day, and the quantity of the subject. We could add entirely new features to images to sculpt and finely control them. We could pick two photos that had a semantic difference, and precisely transfer over the difference by transferring the features. We could also stack hundreds of edits together.

  This really sounds amazing. Did you patch the features from one image to another specifically? Details on how you transferred a subject from one to the other would be appreciated.

• Logan Riggs 19 Jun 2024 15:40 UTC

  LW: 3 AF: 2

  0

  AF

  in reply to: Dan Braun’s comment on: Iden­ti­fy­ing Func­tion­ally Im­por­tant Fea­tures with End-to-End Sparse Dic­tionary Learning

  Thanks so much! All the links and info will save me time:)

  Regarding cos-sim, after thinking a bit, I think it’s more sinister. For cross-cos-sim comparison, you get different results if you take the max over the 0th or 1st dimension (equivalent to doing cos(local, e2e) vs cos(e2e, local). As an example, you could have 2 features each, 3 point in the same direction and 1 points opposte. Making up numbers:

  feature-directions(1D) = [ [1],[1]] & [[1],[-1]]
  cos-sim = [[1, 1], [-1, −1]]

  For more intuition, suppose 4 local features surround 1 e2e feature (and the other features are pointed elsewhere). Then the 4 local features will all have high max-cos sim but the e2e only has 1. So it’s not just double-counting, but quadruple counting. You could see for yourself if you swap your dim=1 to 0 in your code.

  But my original comment showed your results are still directionally correct when doing [global max w/​ replacement] (if I coded it correctly).

  Btw it’s not intuitive to me that the encoder directions might be similar even though the decoder directions are not. Curious if you could share your intuitions here.

  The decoder directions have degrees of freedom, but the encoder directions...might have similar degrees of freedom and I’m wrong, lol. BUT! they might be functionally equivalent, so they activate on similar datapoints across seeds. That is more laborious to check though, waaaah.

  I can check both (encoder directions first) because previous literature is really only on the SVD of gradient (ie the output), but an SAE might be more constrained when separating out inputs into sparse features. Thanks for prompting for my intuition!

• Logan Riggs 19 Jun 2024 15:21 UTC

  3 points

  0

  in reply to: Viliam’s comment on: Suffer­ing Is Not Pain

  You say:

  We cannot realistically expect a significant part of population (let’s say, 10%) to become advanced meditators to the level that they actually become indifferent to pain. So… for practical purposes, “pain causes aversion” describes the situation correctly, for a vast majority of people.

  Which, if this is just a semantic argument, then sure. But OP’s conclusion is goal-oriented:

  Understanding the distinction between pain and suffering is crucial for developing effective strategies to reduce suffering. By directly addressing the craving, aversion, and clinging which cause suffering, we can create more compassionate and impactful interventions.

  When I think of effective strategies here, I think of developing jhana helmets^[1] which would imitate the mind state of blissful-joy-flow state. Although this causes joy-concentrated-collectedness, it’s argued that this puts your mind in a state where it can better notice that aversion/​craving are necessary for suffering (note: I’ve only partially experienced this).

  Although I think you’re expressing skepticism of craving/​aversion as the only necessary cause of suffering for all people? Or maybe just the 99.9999% reduction in suffering (ie the knife) vs a 99% reduction for all? What do you actually believe?

  For me, I read a book that suggested many different experiments to try in a playful way. One was to pay attention to the “distance” between a my current state (e.g. “itching”) and a desired state (“not itching”), and it did feel worse the larger the “distance”. I could even intentionally make it feel larger or small and thought that was very interesting. In one limiting case, you don’t classify the two situations “itching” “not itching” as separate, so no suffering. In the other, it’s “the difference between heaven and hell”, lol. This book had >100 like these (though it is intended for advanced meditators, brag brag).

  1. ^

     those jhana helmet people have pivoted to improving pedagogy w/​ jhana retreats at $1-2k, I think for the purpose of gathering more jhana data, but then it became widely successful

• Logan Riggs 18 Jun 2024 21:58 UTC

  LW: 4 AF: 2

  0

  AF

  on: Iden­ti­fy­ing Func­tion­ally Im­por­tant Fea­tures with End-to-End Sparse Dic­tionary Learning

  The e2e having different feature directions across seeds was quite the bummer, but then I thought “are the encoder directions different though?”
  
  Intuitively the encoder directions affect which datapoints each feature activates on, and the decoder is the causal downstream effect. For e2e, we would expect widely different decoder directions because there are many free parameters (from some other work that showed SVD of gradients had many zero singular values, meaning moving in most directions don’t effect the downstream loss), but not necessarily encoder directions.

  If the encoder directions are similar across seeds, I’d trust them to inform relevant features for the model output (in cases where we don’t care about connections w/​ downstream layers).

  However, I was not able to find the SAEs for various seeds.
  

  Trying to replicate Cos-sim Plots

  I downloaded the similar CE at layer 6 for all three types of SAEs & took their cos-sim (last column in figure 3).

  I think your cos-sim metric gives different results if you take the max over the first or 2nd dimension (or equivalently swapped the order of decoders multiplied by each other). If so, I think this is because you might double-count or something? Regardless, I ended up doing some hungarian algorithm to take the overall max (but don’t double-count), but it’s on cpu, so I only did the first 10k/​40k features. Below is results for both encoder & decoder, which do replicate the directional results.

  • 

  • 

  Nonzero Features

  Additionally I thought that some results were from counting nonzero features, which, for the encoder is some high-cos-sim features, and decoder is the low-cos-sim features weirdly enough.

  • 

  • 

  Would appreciate if y’all upload any repeated seeds!

  My code is temporarily hosted (for a few weeks maybe?) here.

• Logan Riggs 18 Jun 2024 13:57 UTC

  3 points

  0

  in reply to: ntt123’s comment on: Logit Prisms: De­com­pos­ing Trans­former Out­puts for Mechanis­tic Interpretability

  Thanks for the correction! What I meant was figure 7 is better modeled as “these neurons are not monosemantic”since their co-activation has a consistent effect (upweighting 9) which isn’t captured by any individual component, and (I predict) these neurons would do different things on different prompts.

  But I think I see where you’re coming from now, so the above is tangential. You’re just decomposing the logits using previous layers components. So even though intermediate layers logit contribution won’t make any sense (from tuned lens) that’s fine.

  It is interesting in your example of the first two layers counteracting each other. Surely this isn’t true in general, but it could be a common theme of later layers counteracting bigrams (what the embedding is doing?) based off context.

• Logan Riggs 17 Jun 2024 21:35 UTC

  2 points

  0

  in reply to: Raemon’s comment on: Logit Prisms: De­com­pos­ing Trans­former Out­puts for Mechanis­tic Interpretability

  To give my speculation (though I upvoted):

  I believe this work makes sense overall e.g. let’s do logit lens but for individual model components, but does not compare to baselines or mentions SAEs.

  Specifically, what would this method be useful for?

  With logit prisms, we can closely examine how the input embeddings, attention heads, and MLP neurons each contribute to the final output.

  If it’s for isolating which model components are causally responsible for a task (e.g. addition & Q&A), then does it improve on patching in different activations for these different model components (or the linear approximation method Attribution Patching)? In what way?

  Additionally, this post did assume mlp neurons are monosemantic, which isn’t true. This is why we use sparse autoencoders for superposition, which

  A final problem is that the logit attribution with the logit lens doesn’t always work out, as shown by the cited Tuned Lens paper (e.g. directly unembedding early layers usually produces nonsense in which logits are upweighted).

  I did upvote however because I think the standards for a blog post on LW should be lower. Thank you Raemon also for asking for details, because it sucks to get downvoted and not told why.

• Logan Riggs 3 Jun 2024 15:15 UTC

  7 points

  1

  on: Com­ments on An­thropic’s Scal­ing Monosemanticity

  Strong upvote fellow co-author! lol

  Highest-activating Features

  I agree we shouldn’t interpret features by their max-activation, but I think the activation magnitude really does matter. Removing smaller activations affects downstream CE less than larger activations (but this does mean the small activations do matter). A weighted percentage of feature activation captures this more (ie (sum of all golden gate activations)/​(sum of all activations)).

  I do believe “lower-activating examples don’t fit your hypothesis” is bad because of circuits. If you find out that “Feature 3453 is a linear combination of the Golden Gate (GG) feature and the positive sentiment feature” then you do understand this feature at high GG activations, but not low GG + low positive sentiment activations (since you haven’t interpreted low GG activations).

  Your “code-error” feature example is good. If it only fits “code-error” at the largest feature activations & does other things, then if we ablate this feature, we’ll take a capabilities hit because the lower activations were used in other computations. But, let’s focus on the lower activations which we don’t understand are being used in other computations bit. We could also have “code-error” or “deception” being represented in the lower activations of other features which, when co-occurring, cause the model to be deceptive or write code errors.

  [Although, Anthropic showed evidence against this by ablating the code-error feature & running on errored code which predicted a non-error output]

  Finding Features

  Anthropic suggested that if you have a feature that occurs 1/​Billion tokens, you need 1 Billion features. You also mention finding important features. I think SAE’s find features on the dataset you give it. For example, we trained an SAE on only chess data (on a chess-finetuned-Pythia model) & all the features were on chess data. I bet if you trained it on code, it’d find only code features (note: I do think some semantic & token level features that would generalize to other domains).

  Pragmatically, if there are features you care about, then it’s important to train the SAE on many texts that exhibit that feature. This is also true for the safety relevant features.

  In general, I don’t think you need these 1000x feature expansions. Even a 1x feature expansion will give you sparse features (because of the L1 penalty). If you want your model to [have positive personality traits] then you only need to disentangle those features.

  [Note: I think your “SAE’s don’t find all Othello board state features” does not make the point that SAE’s don’t find relevant features, but I’d need to think for 15 min to clearly state it which I don’t want to do now, lol. If you think that’s a crux though, then I’ll try to communicate it]

  Correlated Features

  They said 82% of features had a max of 0.3 correlation which (wait, does this imply that 18% of their million billion features did correlate even more???), I agree is a lot. I think this is strongest evidence for “neuron basis is not as good as SAE’s”, which I’m unsure who is still arguing that, but as a sanity check makes sense.

  However, some neurons are monosemantic so it makes sense for SAE features to also find those (though again, 18% of a milllion billion have a higher correlation than 0.3?)

  > We additionally confirmed that feature activations are not strongly correlated with activations of any residual stream basis direction.

  I’m sure they actually found very strongly correlated features specifically for the outlier dimensions in the residual stream which Anthropic has previous work showing is basis aligned (unless Anthropic trains their models in ways that doesn’t produce an outlier dimension which there is existing lit on).

  [Note: I wrote a lot. Feel free to respond to this comment in parts!]

• Logan Riggs 3 Jun 2024 14:15 UTC

  2 points

  2

  on: How to Bet­ter Re­port Sparse Au­toen­coder Performance

  I think some people use the loss when all features are set to zero, instead of strictly doing

  I think this is an unfinished

• Logan Riggs 17 May 2024 21:06 UTC

  LW: 3 AF: 1

  0

  AF

  on: Iden­ti­fy­ing Func­tion­ally Im­por­tant Fea­tures with End-to-End Sparse Dic­tionary Learning

  What a cool paper! Congrats!:)
  
  What’s cool:
  1. e2e saes learn very different features every seed. I’m glad y’all checked! This seems bad.
  2. e2e SAEs have worse intermediate reconstruction loss than local. I would’ve predicted the opposite actually.
  3. e2e+downstream seems to get all the benefits of the e2e one (same perf at lower L0) at the same compute cost, w/​o the “intermediate activations aren’t similar” problem.

  It looks like you’ve left for future work postraining SAE_local on KL or downstream loss as future work, but that’s a very interesting part! Specifically the approximation of SAE_e2e+downstream as you train on number of tokens.

  Did y’all try ablations on SAE_e2e+downstream? For example, only training on the next layers Reconstruction loss or next N-layers rec loss?

• Logan Riggs 16 Apr 2024 19:13 UTC

  4 points

  0

  on: Ex­per­i­ments with an al­ter­na­tive method to pro­mote spar­sity in sparse autoencoders

  Great work!

  Did you ever run just the L0-approx & sparsity-frequency penalty separately? It’s unclear if you’re getting better results because the L0 function is better or because there are less dead features.

  Also, a feature frequency of 0.2 is very large! ¹⁄₅ tokens activating is large even for positional (because your context length is 128). It’d be bad if the improved results are because polysemanticity is sneaking back in through these activations. Sampling datapoints across a range of activations should show where the meaning becomes polysemantic. Is it the bottom 10% (or 10% of max-activating example is my preferred method)

• Logan Riggs 10 Apr 2024 14:32 UTC

  3 points

  0

  on: Nor­mal­iz­ing Sparse Autoencoders

  For comparing CE-difference (or the mean reconstruction score), did these have similar L0′s? If not, it’s an unfair comparison (higher L0 is usually higher reconstruction accuracy).