Sam Marks

• Sam Marks 26 Apr 2024 0:36 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: Neel Nanda’s comment on: Im­prov­ing Dic­tionary Learn­ing with Gated Sparse Autoencoders

  Yep, you’re totally right—thanks!

• Sam Marks 25 Apr 2024 22:13 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: Arthur Conmy’s comment on: Im­prov­ing Dic­tionary Learn­ing with Gated Sparse Autoencoders

  Oh, one other issue relating to this: in the paper it’s claimed that if $\gamma$ is the argmin of $E[\parallel \hat{x}-{\gamma }^{\prime }x{\parallel }_2^2]$ then $1/\gamma$ is the argmin of $E[\parallel {\gamma }^{\prime }\hat{x}-x\parallel ]$. However, this is not actually true: the argmin of the latter expression is $\frac{E[x\cdot \hat{x}]}{E[\parallel \hat{x}{\parallel }_2^2]}\ne {\left(\frac{E[x\cdot \hat{x}]}{E[\parallel x{\parallel }_2]}\right)}^{-1}$. To get an intuition here, consider the case where $\hat{x}$ and $x$ are very nearly perpendicular, with the angle between them just slightly less than ${90}^{\circ }$. Then you should be able to convince yourself that the best factor to scale either $x$ or $\hat{x}$ by in order to minimize the distance to the other will be just slightly greater than 0. Thus the optimal scaling factors cannot be reciprocals of each other.

  ETA: Thinking on this a bit more, this might actually reflect a general issue with the way we think about feature shrinkage; namely, that whenever there is a nonzero angle between two vectors of the same length, the best way to make either vector close to the other will be by shrinking it. I’ll need to think about whether this makes me less convinced that the usual measures of feature shrinkage are capturing a real thing.

  ETA2: In fact, now I’m a bit confused why your figure 6 shows no shrinkage. Based on what I wrote above in this comment, we should generally expect to see shrinkage (according to the definition given in equation (9)) whenever the autoencoder isn’t perfect. I guess the answer must somehow be “equation (10) actually is a good measure of shrinkage, in fact a better measure of shrinkage than the ‘corrected’ version of equation (10).” That’s pretty cool and surprising, because I don’t really have a great intuition for what equation (10) is actually capturing.

• Sam Marks 25 Apr 2024 21:57 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: Rohin Shah’s comment on: Im­prov­ing Dic­tionary Learn­ing with Gated Sparse Autoencoders

  Ah thanks, you’re totally right—that mostly resolves my confusion. I’m still a little bit dissatisfied, though, because the $L_{\text{aux}}$ term is optimizing for something that we don’t especially want (i.e. for $\hat{x}\left(\text{ReLU}\right({\pi }_{\text{gated}}\left(x\right))$ to do a good job of reconstructing $x$). But I do see how you do need to have some sort of a reconstruction-esque term that actually allows gradients to pass through to the gated network.

• Sam Marks 25 Apr 2024 21:52 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: Sam Marks’s comment on: Im­prov­ing Dic­tionary Learn­ing with Gated Sparse Autoencoders

  (The question in this comment is more narrow and probably not interesting to most people.)

  The limitations section includes this paragraph:

  One worry about increasing the expressivity of sparse autoencoders is that they will overfit when
  reconstructing activations (Olah et al., 2023, Dictionary Learning Worries), since the underlying
  model only uses simple MLPs and attention heads, and in particular lacks discontinuities such as step
  functions. Overall we do not see evidence for this. Our evaluations use held-out test data and we
  check for interpretability manually. But these evaluations are not totally comprehensive: for example,
  they do not test that the dictionaries learned contain causally meaningful intermediate variables in the
  model’s computation. The discontinuity in particular introduces issues with methods like integrated
  gradients (Sundararajan et al., 2017) that discretely approximate a path integral, as applied to SAEs
  by Marks et al. (2024).

  I’m not sure I understand the point about integrated gradients here. I understand this sentence as meaning: since model outputs are a discontinuous function of feature activations, integrated gradients will do a bad job of estimating the effect of patching feature activations to counterfactual values.

  If that interpretation is correct, then I guess I’m confused because I think IG actually handles this sort of thing pretty gracefully. As long as the number of intermediate points you’re using is large enough that you’re sampling points pretty close to the discontinuity on both sides, then your error won’t be too large. This is in contrast to attribution patching which will have a pretty rough time here (but not really that much worse than with the normal ReLU encoders, I guess). (And maybe you also meant for this point to apply to attribution patching?)

• Sam Marks 25 Apr 2024 21:36 UTC

  LW: 4 AF: 3

  2

  AF

  in reply to: Sam Marks’s comment on: Im­prov­ing Dic­tionary Learn­ing with Gated Sparse Autoencoders

  I’m a bit perplexed by the choice of loss function for training GSAEs (given by equation (8) in the paper). The intuitive (to me) thing to do here would be would be to have the $L_{\text{reconstruct}}$ and $L_{\text{sparsity}}$ terms, but not the $L_{\text{aux}}$ term, since the point of ${\pi }_{\text{gate}}$ is to tell you which features should be active, not to itself provide good feature coefficients for reconstructing $x$. I can sort of see how not including this term might result in the coordinates of ${\pi }_{\text{gate}}$ all being extremely small (but barely positive when it’s appropriate to use a feature), such that the sparsity term doesn’t contribute much to the loss. Is that what goes wrong? Are there ablation experiments you can report for this? If so, including this $L_{\text{aux}}$ term still currently seems to me like a pretty unprincipled way to deal with this—can the authors provide any flavor here?

  Here are two ways that I’ve come up with for thinking about this loss function—let me know if either of these are on the right track. Let $f_{\text{gate},\text{ReLU}}$ denote the gated encoder, but with a ReLU activation instead of Heaviside. Note then that $f_{\text{gate},\text{ReLU}}$ is just the standard SAE encoder from Towards Monosemanticity.

  Perspective 1: The usual loss from Towards Monosemanticity for training SAEs is $\parallel x-\hat{x}\left(f_{\text{gate},\text{ReLU}}\right(x\left)\right){\parallel }_2^2+\lambda \parallel f_{\text{gate},\text{ReLU}}\left(x\right){\parallel }_1$ (this is the same as your $L_{\text{sparsity}}$ and $L_{\text{aux}}$ up to the detaching thing). But now you have this magnitude network which needs to get a gradient signal. Let’s do that by adding an additional term $\parallel x-\hat{x}\left(\tilde{f}\right(x\left)\right){\parallel }_2^2$ -- your $L_{\text{reconstruction}}$. So under this perspective, it’s the reconstruction term which is new, with the sparsity and auxiliary terms being carried over from the usual way of doing things.

  Perspective 2 (h/​t Jannik Brinkmann): let’s just add together the usual Towards Monosemanticity loss function for both the usual architecture and the new modified archiecture: $L=L_{\text{reconstruction}}\left(\tilde{f}\right)+L_{\text{reconstruction}}\left(\tilde{f}\right)+L_{\text{sparsity}}\left(f_{\text{gate},\text{ReLU}}\right)+L_{\text{sparsity}}\left(f_{\text{gate},\text{ReLU}}\right)$.

  However, the gradients with respect to the second term in this sum vanish because of the use of the Heaviside, so the gradient with respect to this loss is the same as the gradient with respect to the loss you actually used.

• Sam Marks 25 Apr 2024 21:12 UTC

  LW: 9 AF: 6

  3

  AF

  in reply to: Sam Marks’s comment on: Im­prov­ing Dic­tionary Learn­ing with Gated Sparse Autoencoders

  I believe that equation (10) giving the analytical solution to the optimization problem defining the relative reconstruction bias is incorrect. I believe the correct expression should be $\gamma =E_{x\sim D}\left[\frac{\hat{x}\cdot x}{\parallel x{\parallel }_2^2}\right]$.

  You could compute this by differentiating equation (9), setting it equal to 0 and solving for $\gamma$. But here’s a more geometrical argument.

  

  By definition, $\gamma x$ is the multiple of $x$ closest to $\hat{x}$. Equivalently, this closest such vector can be described as the projection ${proj}_x\left(\hat{x}\right)=\frac{\hat{x}\cdot x}{\parallel x{\parallel }_2^2}x$. Setting these equal, we get the claimed expression for $\gamma$.

  As a sanity check, when our vectors are 1-dimensional, $x=1$, and $\hat{x}=\frac{1}{2}$, we my expression gives $\gamma =\frac{1}{2}$ (which is correct), but equation (10) in the paper gives $\frac{1}{\sqrt{3}}$.

• Sam Marks 25 Apr 2024 20:55 UTC

  LW: 17 AF: 12

  2

  AF

  on: Im­prov­ing Dic­tionary Learn­ing with Gated Sparse Autoencoders

  Great work! Obviously the results here speak for themselves, but I especially wanted to complement the authors on the writing. I thought this paper was a pleasure to read, and easily a top 5% exemplar of clear technical writing. Thanks for putting in the effort on that.

  I’ll post a few questions as children to this comment.

• Sam Marks 22 Apr 2024 20:23 UTC

  LW: 4 AF: 2

  0

  AF

  in reply to: Fabien Roger’s comment on: Discrim­i­nat­ing Be­hav­iorally Iden­ti­cal Clas­sifiers: a model prob­lem for ap­ply­ing in­ter­pretabil­ity to scal­able oversight

  I’m pretty sure that you’re not correct that the interpretation step from our SHIFT experiments essentially relies on using data from the Pile. I strongly expect that if we were to only use inputs from $D_a$ then we would be able to interpret the SAE features about as well. E.g. some of the SAE features only activate on female pronouns, and we would be able to notice this. Technically, we wouldn’t be able to rule out the hypothesis “this feature activates on female pronouns only when their antecedent is a nurse,” but that would be a bit of a crazy hypothesis anyway.

  In more realistic settings (larger models and subtler behaviors) we might have more serious problems ruling out hypotheses like this. But I don’t see any fundamental reason that using disambiguating datapoints is strictly necessary.

• Sam Marks 20 Apr 2024 21:11 UTC

  LW: 2 AF: 1

  0

  AF

  in reply to: Sam Marks’s comment on: Discrim­i­nat­ing Be­hav­iorally Iden­ti­cal Clas­sifiers: a model prob­lem for ap­ply­ing in­ter­pretabil­ity to scal­able oversight

  (Edits made. In the edited version, I think the only questionable things are the title and the line “[In this post, I will a]rticulate a class of approaches to scalable oversight I call cognition-based oversight.” Maybe I should be even more careful and instead say that cognition-based oversight is merely something that “could be useful for scalable oversight,” but I overall feel okay about this.

  Everywhere else, I think the term “scalable oversight” is now used in the standard way.)

• Sam Marks 20 Apr 2024 20:52 UTC

  LW: 6 AF: 5

  2

  AF

  in reply to: Buck’s comment on: Discrim­i­nat­ing Be­hav­iorally Iden­ti­cal Clas­sifiers: a model prob­lem for ap­ply­ing in­ter­pretabil­ity to scal­able oversight

  I (mostly; see below) agree that in this post I used the term “scalable oversight” in a way which is non-standard and, moreover, in conflict way the way I typically use the term personally. I also agree with the implicit meta-point that it’s important to be careful about using terminology in a consistent way (though I probably don’t think it’s as important as you do). So overall, after reading this comment, I wish I had been more careful about how I treated the term “scalable oversight.” After I post this comment, I’ll make some edits for clarity, but I don’t expect to go so far as to change the title^[1].

  Two points in my defense:

  1. Even though “scalable oversight” isn’t an appropriate description for the narrow technical problem I pose here, the way I expect progress on this agenda to actually get applied is well-described as scalable oversight.

  2. I’ve found the scalable oversight frame on this problem useful both for my own thinking about it and for explaining it to others.

  Re (1): I spend most of my time thinking about the sycophantic reward hacking threat model. So in my head, some of the model’s outputs really are bad but it’s hard to notice this. Here are two ways that I think this agenda could help with noticing bad particular outputs:

  1. By applying DBIC to create classifiers for particular bad things (e.g. measurement tampering) which we apply to detect bad outputs.

  2. By giving us a signal about which episodes should be more closely scrutinized, and which aspects of those episodes we should scrutinize. (For example, suppose you notice that your model is thinking about a particular camera in a maybe-suspicious way, so you look for tricky ways that that camera could have been tampered with, and after a bunch of targeted scrutiny you notice a hack).

  I think that both of these workflows are accurately described as scalable oversight.

  Re (2): when I explain that I want to apply interpretability to scalable oversight, people—including people that I really expected to know better—often react with surprise. This isn’t, I think, because they’re thinking carefully about what scalable oversight means the way that you are. Rather, it seems that a lot of people split alignment work into two non-interacting magisteria called “scalable oversight” and “solving deceptive alignment,” and they classify interpretability work as being part of the latter magisterium. Such people tend to not realize that e.g. ELK is centrally a scalable oversight agenda, and I think of my proposed agenda here as attempting to make progress on ELK (or on special cases thereof).

  I guess my post muddies the water on all of the above by bringing up scheming; even though this technically fits into the setting I propose to make progress on, I don’t really view it as the central problem I’m trying to solve.

  1. ^

     Sadly, if I say that my goal is to use interpretability to “evaluate models” then I think people will pattern-match this to “evals” which typically means something different, e.g. checking for dangerous capabilities. I can’t really think of a better, non-confusing term for the task of “figuring out whether a model is good or bad.” Also, I expect that the ways progress on this agenda will actually be applied do count as “scalable oversight”; see below.

• Sam Marks 19 Apr 2024 20:54 UTC

  LW: 2 AF: 1

  0

  AF

  on: [Full Post] Progress Up­date #1 from the GDM Mech In­terp Team

  With the ITO experiments, my first guess would be that reoptimizing the sparse approximation problem is mostly relearning the encoder, but with some extra uninterpretable hacks for low activation levels that happen to improve reconstruction. In other words, I’m guessing that the boost in reconstruction accuracy (and therefore loss recovered) is mostly not due to better recognizing the presence of interpretable features, but by doing fiddly uninterpretable things at low activation levels.

  I’m not really sure how to operationalize this into a prediction. Maybe something like: if you pick some small-ish threshold T (maybe like T=3 based on the plot copied below) and round activations less than T down to 0 (for both the ITO encoder and the original encoder), then you’ll no longer see that the ITO encoder outperforms the original one.

  

• Sam Marks 19 Apr 2024 20:46 UTC

  LW: 9 AF: 7

  0

  AF

  on: [Full Post] Progress Up­date #1 from the GDM Mech In­terp Team

  Awesome stuff—I think that updates like this (both from the GDM team and from Anthropic) are very useful for organizing work in this space. And I especially appreciate the way this was written, with both short summaries and in-depth write-ups.

Discrim­i­nat­ing Be­hav­iorally Iden­ti­cal Clas­sifiers: a model prob­lem for ap­ply­ing in­ter­pretabil­ity to scal­able oversight

• Sam Marks 25 Feb 2024 20:05 UTC

  2 points

  0

  in reply to: mishajw’s comment on: How well do truth probes gen­er­al­ise?

  I originally ran some of these experiments on 7B and got very different results, that PCA plot of 7B looks familiar (and bizarre).

  I found that the PCA plot for 7B for larger_than and smaller_than individually looked similar to that for 13B, but that the PCA plot for larger_than + smaller_than looked degenerate in the way I screenshotted. Are you saying that your larger_than + smaller_than PCA looked familiar for 7B?

  I suppose there are two things we want to separate: “truth” from likely statements, and “truth” from what humans think (under some kind of simulacra framing). I think this approach would allow you to do the former, but not the latter. And to be honest, I’m not confident on TruthfulQA’s ability to do the latter either.

  Agreed on both points.

  We differ slightly from the original GoT paper in naming, and use got_cities to refer to both the cities and neg_cities datasets. The same is true for sp_en_trans and larger_than. We don’t do this for cities_cities_{conj,disj} and leave them unpaired.

  Thanks for clarifying! I’m guessing this is what’s making the GoT datasets much worse for generalization (from and to) in your experiments. For 13B, it mostly seemed to me that training on negated statements helped for generalization to other negated statements, and that pairing negated statements with unnegated statements in training data usually (but not always) made generalization to unnegated datasets a bit worse. (E.g. the cities → sp_en_trans generalization is better than cities + neg_cities → sp_en_trans generalization.)

• Sam Marks 25 Feb 2024 13:40 UTC

  4 points

  1

  on: How well do truth probes gen­er­al­ise?

  Very cool! Always nice to see results replicated and extended on, and I appreciated how clear you were in describing your experiments.

  Do smaller models also have a generalised notion of truth?

  In my most recent revision of GoT^[1] we did some experiments to see how truth probe generalization changes with model scale, working with LLaMA-2-7B, −13B, and −70B. Result: truth probes seems to generalize better for larger models. Here are the relevant figures.

  

  

  Some other related evidence from our visualizations:

  

  We summed things up like so, which I’ll just quote in its entirety:

  Overall, these visualizations suggest a picture like the following: as LLMs scale (and perhaps, also as a fixed LLM progresses through its forward pass), they hierarchically develop and linearly represent increasingly general abstractions. Small models represent surface-level characteristics of their inputs; these surface-level characteristics may be sufficient for linear probes to be accurate on narrow training distributions, but such probes are unlikely to generalize out-of-distribution. Large models linearly represent more abstract concepts, potentially including abstract notions like “truth” which capture shared properties of topically and structurally diverse inputs. In middle regimes, we may find linearly represented concepts of intermediate levels of abstraction, for example, “accurate factual recall” or “close association” (in the sense that “Beijing” and “China” are closely associated). These concepts may suffice to distinguish true/​false statements on individual datasets, but will only generalize to test data for which the same concepts
  suffice.

  How do we know we’re detecting truth, and not just likely statements?

  One approach here is to use a dataset in which the truth and likelihood of inputs are uncorrelated (or negatively correlated), as you kinda did with TruthfulQA. For that, I like to use the “neg_” versions of the datasets from GoT, containing negated statements like “The city of Beijing is not in China.” For these datasets, the correlation between truth value and likelihood (operationalized as LLaMA-2-70B’s log probability of the full statement) is strong and negative (-0.63 for neg_cities and -.89 for neg_sp_en_trans). But truth probes still often generalize well to these negated datsets. Here are results for LLaMA-2-70B (the horizontal axis shows the train set, and the vertical axis shows the test set).

  

  We also find that the probe performs better than LDA in-distribution, but worse out-of-distribution:

  Yep, we found the same thing—LDA improves things in-distribution, but generalizes work than simple DIM probes.

  Why does got_cities_cities_conj generalise well?

  I found this result surprising, thanks! I don’t really have great guesses for what’s going on. One thing I’ll say is that it’s worth tracking differences between various sorts of factual statements. For example, for LLaMA-2-13B it generally seemed to me that there was better probe transfer between factual recall datasets (e.g. cities and sp_en_trans, but not larger_than). I’m not really sure why the conjunctions are making things so much better, beyond possibly helping to narrow down on “truth” beyond just “correct statement of factual recall.”

  I’m not surprised that cities_cities_conj and cities_cities_disj are so qualitatively different—cities_cities_disj has never empirically played well with the other datasets (in the sense of good probe transfer) and I don’t really know why.

  1. ^

     This is currently under review, but not yet on arxiv, sorry about that! Code in the nnsight branch here. I’ll try to come back to add a link to the paper once I post it or it becomes publicly available on OpenReview, whichever happens first.

• Sam Marks 9 Feb 2024 23:47 UTC

  2 points

  0

  in reply to: Joseph Bloom’s comment on: Open Source Sparse Au­toen­coders for all Resi­d­ual Stream Lay­ers of GPT2-Small

  This comment is about why we were getting different MSE numbers. The answer is (mostly) benign—a matter of different scale factors. My parallel comment, which discusses why we were getting different CE diff numbers is the more important one.

  When you compute MSE loss between some activations $x$ and their reconstruction $\hat{x}$, you divide by variance of $x$, as estimated from the data in a batch. I’ll note that this doesn’t seem like a great choice to me. Looking at the resulting training loss:

  $\parallel x-\hat{x}{\parallel }_2^2/Var\left(x\right)+\lambda \parallel f{\parallel }_1$

  where $f$ is the encoding of $x$ by the autoencoder and $\lambda$ is the L1 regularization constant, we see that if you scale $x$ by some constant $\alpha$, this will have no effect on the first term, but will scale the second term by $\alpha$. So if activations generically become larger in later layers, this will mean that the sparsity term becomes automatically more strongly weighted.

  I think a more principled choice would be something like

  $\parallel x-\hat{x}{\parallel }_2+\lambda \parallel f{\parallel }_1$

  where we’re no longer normalizing by the variance, and are also using sqrt(MSE) instead of MSE. (This is what the dictionary_learning repo does.) When you scale $x$ by a constant $\alpha$, this entire expression scales by a factor of $\alpha$, so that the balance between reconstruction and sparsity remains the same. (On the other hand, this will mean that you might need to scale the learning rate by $1/\alpha$, so perhaps it would be reasonable to divide through this expression by $\parallel x{\parallel }_2$? I’m not sure.)

  ---

  Also, one other thing I noticed: something which we both did was to compute MSE by taking the mean over the squared difference over the batch dimension and the activation dimension. But this isn’t quite what MSE usually means; really we should be summing over the activation dimension and taking the mean over the batch dimension. That means that both of our MSEs are erroneously divided by a factor of the hidden dimension (768 for you and 512 for me).

  This constant factor isn’t a huge deal, but it does mean that:

  1. The MSE losses that we’re reporting are deceptively low, at least for the usual interpretation of “mean squared error”

  2. If we decide to fix this, we’ll need to both scale up our L1 regularization penalty by a factor of the hidden dimension (and maybe also scale down the learning rate).

  This is a good lesson on how MSE isn’t naturally easy to interpret and we should maybe just be reporting percent variance explained. But if we are going to report MSE (which I have been), I think we should probably report it according to the usual definition.

• Sam Marks 9 Feb 2024 23:18 UTC

  9 points

  2

  in reply to: Joseph Bloom’s comment on: Open Source Sparse Au­toen­coders for all Resi­d­ual Stream Lay­ers of GPT2-Small

  Yep, as you say, @Logan Riggs figured out what’s going on here: you evaluated your reconstruction loss on contexts of length 128, whereas I evaluated on contexts of arbitrary length. When I restrict to context length 128, I’m able to replicate your results.

  Here’s Logan’s plot for one of your dictionaries (not sure which)

  

  and here’s my replication of Logan’s plot for your layer 1 dictionary

  

  Interestingly, this does not happen for my dictionaries! Here’s the same plot but for my layer 1 residual stream output dictionary for pythia-70m-deduped

  

  (Note that all three plots have a different y-axis scale.)

  Why the difference? I’m not really sure. Two guesses:

  1. The model: GPT2-small uses learned positional embeddings whereas Pythia models use rotary embeddings

  2. The training: I train my autoencoders on variable-length sequences up to length 128; left padding is used to pad shorter sequences up to length 128. Maybe this makes a difference somehow.

  ---

  In terms of standardization of which metrics to report, I’m torn. On one hand, for the task your dictionaries were trained on (reconstruction activations taken from length 128 sequences), they’re performing well and this should be reflected in the metrics. On the other hand, people should be aware that if they just plug your autoencoders into GPT2-small and start doing inference on inputs found in the wild, things will go off the rails pretty quickly. Maybe the answer is that CE diff should be reported both for sequences of the same length used in training and for arbitrary-length sequences?

  What links here?

  • Ex­am­in­ing Lan­guage Model Perfor­mance with Re­con­structed Ac­ti­va­tions us­ing Sparse Au­toen­coders by Evan Anders (27 Feb 2024 2:43 UTC; 41 points)

• Sam Marks 7 Feb 2024 22:42 UTC

  2 points

  0

  in reply to: Joseph Bloom’s comment on: Open Source Sparse Au­toen­coders for all Resi­d­ual Stream Lay­ers of GPT2-Small

  My SAEs also have a tied decoder bias which is subtracted from the original activations. Here’s the relevant code in dictionary.py

  def encode(self, x):
          return nn.ReLU()(self.encoder(x - self.bias))
      
      def decode(self, f):
          return self.decoder(f) + self.bias
      
      def forward(self, x, output_features=False, ghost_mask=None):
              [...]
              f = self.encode(x)
              x_hat = self.decode(f)
              [...]
              return x_hat
  

  Note that I checked that our SAEs have the same input-output behavior in my linked colab notebook. I think I’m a bit confused why subtracting off the decoder bias had to be done explicitly in your code—maybe you used dictionary.encoder and dictionary.decoder instead of dictionary.encode and dictionary.decode? (Sorry, I know this is confusing.) ETA: Simple things I tried based on the hypothesis “one of us needs to shift our inputs by +/​- the decoder bias” only made things worse, so I’m pretty sure that you had just initially converted my dictionaries into your infrastructure in a way that messed up the initial decoder bias, and therefore had to hand-correct it.

  I note that the MSE Loss you reported for my dictionary actually is noticeably better than any of the MSE losses I reported for my residual stream dictionaries! Which layer was this? Seems like something to dig into.

• Sam Marks 7 Feb 2024 22:33 UTC

  3 points

  1

  in reply to: Arthur Conmy’s comment on: Some open-source dic­tio­nar­ies and dic­tio­nary learn­ing infrastructure

  At the time that I made this post, no, but this has been implemented in dictionary_learning since I saw your suggestion to do so in your linked post.

• Sam Marks 7 Feb 2024 1:52 UTC

  2 points

  0

  in reply to: Sam Marks’s comment on: Open Source Sparse Au­toen­coders for all Resi­d­ual Stream Lay­ers of GPT2-Small

  Another sanity check: when you compute CE loss using the same code that you use when computing CE loss when activations are reconstructed by the autoencoders, but instead of actually using the autoencoder you just plug the correct activations back in, do you get the same answer (~3.3) as when you evaluate CE loss normally?