Sam Marks

Karma: 1,636

Sam Marks 25 Feb 2024 20:05 UTC
2 points
0
in reply to: mishajw’s comment on: How well do truth probes generalise?
I originally ran some of these experiments on 7B and got very different results, that PCA plot of 7B looks familiar (and bizarre).
I found that the PCA plot for 7B for larger_than and smaller_than individually looked similar to that for 13B, but that the PCA plot for larger_than + smaller_than looked degenerate in the way I screenshotted. Are you saying that your larger_than + smaller_than PCA looked familiar for 7B?
I suppose there are two things we want to separate: “truth” from likely statements, and “truth” from what humans think (under some kind of simulacra framing). I think this approach would allow you to do the former, but not the latter. And to be honest, I’m not confident on TruthfulQA’s ability to do the latter either.
Agreed on both points.
We differ slightly from the original GoT paper in naming, and use got_cities to refer to both the cities and neg_cities datasets. The same is true for sp_en_trans and larger_than. We don’t do this for cities_cities_{conj,disj} and leave them unpaired.
Thanks for clarifying! I’m guessing this is what’s making the GoT datasets much worse for generalization (from and to) in your experiments. For 13B, it mostly seemed to me that training on negated statements helped for generalization to other negated statements, and that pairing negated statements with unnegated statements in training data usually (but not always) made generalization to unnegated datasets a bit worse. (E.g. the cities → sp_en_trans generalization is better than cities + neg_cities → sp_en_trans generalization.)

Sam Marks 25 Feb 2024 13:40 UTC
4 points
1
on: How well do truth probes generalise?
Very cool! Always nice to see results replicated and extended on, and I appreciated how clear you were in describing your experiments.
Do smaller models also have a generalised notion of truth?
In my most recent revision of GoT^[1] we did some experiments to see how truth probe generalization changes with model scale, working with LLaMA-2-7B, −13B, and −70B. Result: truth probes seems to generalize better for larger models. Here are the relevant figures.
Some other related evidence from our visualizations:
We summed things up like so, which I’ll just quote in its entirety:
Overall, these visualizations suggest a picture like the following: as LLMs scale (and perhaps, also as a fixed LLM progresses through its forward pass), they hierarchically develop and linearly represent increasingly general abstractions. Small models represent surface-level characteristics of their inputs; these surface-level characteristics may be sufficient for linear probes to be accurate on narrow training distributions, but such probes are unlikely to generalize out-of-distribution. Large models linearly represent more abstract concepts, potentially including abstract notions like “truth” which capture shared properties of topically and structurally diverse inputs. In middle regimes, we may find linearly represented concepts of intermediate levels of abstraction, for example, “accurate factual recall” or “close association” (in the sense that “Beijing” and “China” are closely associated). These concepts may suffice to distinguish true/false statements on individual datasets, but will only generalize to test data for which the same concepts
suffice.
How do we know we’re detecting truth, and not just likely statements?
One approach here is to use a dataset in which the truth and likelihood of inputs are uncorrelated (or negatively correlated), as you kinda did with TruthfulQA. For that, I like to use the “neg_” versions of the datasets from GoT, containing negated statements like “The city of Beijing is not in China.” For these datasets, the correlation between truth value and likelihood (operationalized as LLaMA-2-70B’s log probability of the full statement) is strong and negative (-0.63 for neg_cities and -.89 for neg_sp_en_trans). But truth probes still often generalize well to these negated datsets. Here are results for LLaMA-2-70B (the horizontal axis shows the train set, and the vertical axis shows the test set).
We also find that the probe performs better than LDA in-distribution, but worse out-of-distribution:
Yep, we found the same thing—LDA improves things in-distribution, but generalizes work than simple DIM probes.
Why does got_cities_cities_conj generalise well?
I found this result surprising, thanks! I don’t really have great guesses for what’s going on. One thing I’ll say is that it’s worth tracking differences between various sorts of factual statements. For example, for LLaMA-2-13B it generally seemed to me that there was better probe transfer between factual recall datasets (e.g. cities and sp_en_trans, but not larger_than). I’m not really sure why the conjunctions are making things so much better, beyond possibly helping to narrow down on “truth” beyond just “correct statement of factual recall.”
I’m not surprised that cities_cities_conj and cities_cities_disj are so qualitatively different—cities_cities_disj has never empirically played well with the other datasets (in the sense of good probe transfer) and I don’t really know why.
1. ^
  This is currently under review, but not yet on arxiv, sorry about that! Code in the nnsight branch here. I’ll try to come back to add a link to the paper once I post it or it becomes publicly available on OpenReview, whichever happens first.

Sam Marks 9 Feb 2024 23:47 UTC
2 points
0
in reply to: Joseph Bloom’s comment on: Open Source Sparse Autoencoders for all Residual Stream Layers of GPT2-Small
This comment is about why we were getting different MSE numbers. The answer is (mostly) benign—a matter of different scale factors. My parallel comment, which discusses why we were getting different CE diff numbers is the more important one.
When you compute MSE loss between some activations $x$ and their reconstruction $^x$ , you divide by variance of $x$ , as estimated from the data in a batch. I’ll note that this doesn’t seem like a great choice to me. Looking at the resulting training loss:
$∥ x -^x ∥_{2}^{2} / V a r (x) + λ ∥ f ∥_{1}$
where $f$ is the encoding of $x$ by the autoencoder and $λ$ is the L1 regularization constant, we see that if you scale $x$ by some constant $α$ , this will have no effect on the first term, but will scale the second term by $α$ . So if activations generically become larger in later layers, this will mean that the sparsity term becomes automatically more strongly weighted.
I think a more principled choice would be something like
$∥ x -^x ∥_{2} + λ ∥ f ∥_{1}$
where we’re no longer normalizing by the variance, and are also using sqrt(MSE) instead of MSE. (This is what the dictionary_learning repo does.) When you scale $x$ by a constant $α$ , this entire expression scales by a factor of $α$ , so that the balance between reconstruction and sparsity remains the same. (On the other hand, this will mean that you might need to scale the learning rate by $1 / α$ , so perhaps it would be reasonable to divide through this expression by $∥ x ∥_{2}$ ? I’m not sure.)
Also, one other thing I noticed: something which we both did was to compute MSE by taking the mean over the squared difference over the batch dimension and the activation dimension. But this isn’t quite what MSE usually means; really we should be summing over the activation dimension and taking the mean over the batch dimension. That means that both of our MSEs are erroneously divided by a factor of the hidden dimension (768 for you and 512 for me).
This constant factor isn’t a huge deal, but it does mean that:
1. The MSE losses that we’re reporting are deceptively low, at least for the usual interpretation of “mean squared error”
2. If we decide to fix this, we’ll need to both scale up our L1 regularization penalty by a factor of the hidden dimension (and maybe also scale down the learning rate).
This is a good lesson on how MSE isn’t naturally easy to interpret and we should maybe just be reporting percent variance explained. But if we are going to report MSE (which I have been), I think we should probably report it according to the usual definition.

Sam Marks 9 Feb 2024 23:18 UTC
9 points
2
in reply to: Joseph Bloom’s comment on: Open Source Sparse Autoencoders for all Residual Stream Layers of GPT2-Small
Yep, as you say, @Logan Riggs figured out what’s going on here: you evaluated your reconstruction loss on contexts of length 128, whereas I evaluated on contexts of arbitrary length. When I restrict to context length 128, I’m able to replicate your results.
Here’s Logan’s plot for one of your dictionaries (not sure which)
and here’s my replication of Logan’s plot for your layer 1 dictionary
Interestingly, this does not happen for my dictionaries! Here’s the same plot but for my layer 1 residual stream output dictionary for pythia-70m-deduped
(Note that all three plots have a different y-axis scale.)
Why the difference? I’m not really sure. Two guesses:
1. The model: GPT2-small uses learned positional embeddings whereas Pythia models use rotary embeddings
2. The training: I train my autoencoders on variable-length sequences up to length 128; left padding is used to pad shorter sequences up to length 128. Maybe this makes a difference somehow.
In terms of standardization of which metrics to report, I’m torn. On one hand, for the task your dictionaries were trained on (reconstruction activations taken from length 128 sequences), they’re performing well and this should be reflected in the metrics. On the other hand, people should be aware that if they just plug your autoencoders into GPT2-small and start doing inference on inputs found in the wild, things will go off the rails pretty quickly. Maybe the answer is that CE diff should be reported both for sequences of the same length used in training and for arbitrary-length sequences?
What links here?
- Examining Language Model Performance with Reconstructed Activations using Sparse Autoencoders by Evan Anders (27 Feb 2024 2:43 UTC; 39 points)

Sam Marks 7 Feb 2024 22:42 UTC
2 points
0
in reply to: Joseph Bloom’s comment on: Open Source Sparse Autoencoders for all Residual Stream Layers of GPT2-Small
My SAEs also have a tied decoder bias which is subtracted from the original activations. Here’s the relevant code in dictionary.py
```
def encode(self, x):
        return nn.ReLU()(self.encoder(x - self.bias))
    
    def decode(self, f):
        return self.decoder(f) + self.bias
    
    def forward(self, x, output_features=False, ghost_mask=None):
            [...]
            f = self.encode(x)
            x_hat = self.decode(f)
            [...]
            return x_hat
```
Note that I checked that our SAEs have the same input-output behavior in my linked colab notebook. I think I’m a bit confused why subtracting off the decoder bias had to be done explicitly in your code—maybe you used dictionary.encoder and dictionary.decoder instead of dictionary.encode and dictionary.decode? (Sorry, I know this is confusing.) ETA: Simple things I tried based on the hypothesis “one of us needs to shift our inputs by +/- the decoder bias” only made things worse, so I’m pretty sure that you had just initially converted my dictionaries into your infrastructure in a way that messed up the initial decoder bias, and therefore had to hand-correct it.

I note that the MSE Loss you reported for my dictionary actually is noticeably better than any of the MSE losses I reported for my residual stream dictionaries! Which layer was this? Seems like something to dig into.

Sam Marks 7 Feb 2024 22:33 UTC
3 points
1
in reply to: Arthur Conmy’s comment on: Some open-source dictionaries and dictionary learning infrastructure
At the time that I made this post, no, but this has been implemented in dictionary_learning since I saw your suggestion to do so in your linked post.

Sam Marks 7 Feb 2024 1:52 UTC
2 points
0
in reply to: Sam Marks’s comment on: Open Source Sparse Autoencoders for all Residual Stream Layers of GPT2-Small
Another sanity check: when you compute CE loss using the same code that you use when computing CE loss when activations are reconstructed by the autoencoders, but instead of actually using the autoencoder you just plug the correct activations back in, do you get the same answer (~3.3) as when you evaluate CE loss normally?

Sam Marks 7 Feb 2024 1:34 UTC
2 points
0
in reply to: Joseph Bloom’s comment on: Open Source Sparse Autoencoders for all Residual Stream Layers of GPT2-Small
In the notebook I link in my original comment, I check that the activations I get out of nnsight are the same as the activations that come from transformer_lens. Together with the fact that our sparsity statistics broadly align, I’m guessing that the issue isn’t that I’m extracting different activations than you are.

Repeating my replication attempt with data from OpenWebText, I get this:

Layer MSE Loss % Variance Explained L1 L0 % Alive CE Reconstructed

1 0.069 95 40 15 46 6.45

7 0.81 86 125 59.2 96 4.38

Broadly speaking, same story as above, except that the MSE losses look better (still not great), and that the CE reconstructed looks very bad for layer 1.

I don’t much padding at all, that might be a big difference too.

Seems like there was a typo here—what do you mean?

Logan Riggs reports that he tried to replicate your results and got something more similar to you. I think Logan is making decisions about padding and tokenization more like the decisions you make, so it’s possible that the difference is down to something around padding and tokenization.

Possible next steps:
- Can you report your MSE Losses (instead of just variance explained)?
- Can you try to evaluate the residual stream dictionaries in the 5_32768 set released here? If you get CE reconstructed much better than mine, then it means that we’re computing CE reconstructed in different ways, where your way consistently reports better numbers. If you get CE reconstructed much worse than mine, then it might mean that there’s a translation error between our codebases (e.g. using different activations).

Layer	MSE Loss	% Variance Explained	L1	L0	% Alive	CE Reconstructed
1	0.069	95	40	15	46	6.45
7	0.81	86	125	59.2	96	4.38

Sam Marks 6 Feb 2024 7:55 UTC
LW: 8 AF: 3
2
AF
on: Open Source Sparse Autoencoders for all Residual Stream Layers of GPT2-Small
I tried replicating your statistics using my own evaluation code (in evaluation.py here). I pseudo-randomly chose layer 1 and layer 7. Sadly, my results look rather different from yours:

Layer MSE Loss % Variance Explained L1 L0 % Alive CE Reconstructed

1 0.11 92 44 17.5 54 5.95

7 1.1 82 137 65.4 95 4.29

Places where our metrics agree: L1 and L0.

Places where our metrics disagree, but probably for a relatively benign reason:
- Percent variance explained: my numbers are slightly lower than yours, and from a brief skim of your code I think it’s because you’re calculating variance slightly incorrectly: you’re not subtracting off the activation’s mean before doing .pow(2).sum(-1). This will slightly overestimate the variance of the original activations, so probably also overestimate percent variance explained.
- Percent alive: my numbers are slightly lower than yours, and this is probably because I determined whether neurons are alive on a (somewhat small) batch of 8192 tokens. So my number is probably an underestimate and yours is correct.
Our metrics disagree strongly on CE reconstructed, and this is a bit alarming. It means that either you have a bug which significantly underestimates reconstructed CE loss, or I have a bug which significantly overestimates it. I think I’m ⁵⁰⁄₅₀ on which it is. Note that according to my stats, your MSE loss is kinda bad, which would suggest that you should also have high CE reconstructed (especially when working with residual stream dictionaries! (in contrast to e.g. MLP dictionaries which are much more forgiving)).

Spitballing a possible cause: when computing CE loss, did you exclude padding tokens? If not, then it’s possible that many of the tokens on which you’re computing CE are padding tokens, which is artificially making your CE look extremely good.

Here is my code. You’ll need to pip install nnsight before running it. Many thanks to Caden Juang for implementing the UnifiedTransformer functionality in nnsight, which is a crazy Frankenstein marriage of nnsight and transformer_lens; it would have been very hard for me to attempt this replication without this feature.

Layer	MSE Loss	% Variance Explained	L1	L0	% Alive	CE Reconstructed
1	0.11	92	44	17.5	54	5.95
7	1.1	82	137	65.4	95	4.29

Sam Marks 6 Feb 2024 6:56 UTC
LW: 7 AF: 4
AF
on: Sam Marks’s Shortform
Some updates about the dictionary_learning repo:
- The repo now has support for ghost grads. h/t g-w1 for submitting a PR for this
- ActivationBuffers now work natively with model components—like the residual stream—whose activations are typically returned as tuples; the buffer knows to take the first component of the tuple (and will iteratively do this if working with nested tuples).
- ActivationBuffers can now be stored on the GPU.
- The file evaluation.py contains code for evaluating trained dictionaries. I’ve found this pretty useful for quickly evaluating dictionaries people send to me.
- New convenience: you can do reconstructed_acts, features = dictionary(acts, output_features=True) to get both the reconstruction and the features computed by dictionary.
Also, if you’d like to train dictionaries for many model components in parallel, you can use the parallel branch. I don’t promise to never make breaking changes to the parallel branch, sorry.
Finally, we’ve released a new set of dictionaries for the MLP outputs, attention outputs, and residual stream in all layers of Pythia-70m-deduped. The MLP and attention dictionaries seem pretty good, and the residual stream dictionaries seem like a mixed bag. Their stats can be found here.

Sam Marks 20 Jan 2024 2:29 UTC
6 points
0
in reply to: TurnTrout’s comment on: TurnTrout’s shortform feed
Thanks, I’ve disliked the shoggoth meme for a while, and this post does a better job articulating why than I’ve been able to do myself.

Sam Marks 12 Jan 2024 19:07 UTC
LW: 4 AF: 3
0
AF
in reply to: Rohin Shah’s comment on: What’s up with LLMs representing XORs of arbitrary features?
Imo “true according to Alice” is nowhere near as “crazy” a feature as “has_true XOR has_banana”. It seems useful for the LLM to model what is true according to Alice! (Possibly I’m misunderstanding what you mean by “crazy” here.)
I agree with this! (And it’s what I was trying to say; sorry if I was unclear.) My point is that
{ features which are as crazy as “true according to Alice” (i.e., not too crazy)}
seems potentially manageable, where as
{ features which are as crazy as arbitrary boolean functions of other features }
seems totally unmanageable.
Thanks, as always, for the thoughtful replies.

Sam Marks 12 Jan 2024 0:12 UTC
LW: 2 AF: 1
0
AF
in reply to: Rohin Shah’s comment on: What’s up with LLMs representing XORs of arbitrary features?
Idk, I think it’s pretty hard to know what things are and aren’t useful for predicting the next token. For example, some of your features involve XORing with a “has_not” feature—XORing with an indicator for “not” might be exactly what you want to do to capture the effect of the “not”.
I agree that “the model has learned the algorithm ‘always compute XORs with has_not’” is a pretty sensible hypothesis. (And might be useful to know, if true!) FWIW, the stronger example of “clearly not useful XORs” I was thinking of has_true XOR has_banana, where I’m guessing you’re anticipating that this XOR exists incidentally.
If you want you could rephrase this issue as ” $a$ and $a \oplus b$ are spuriously correlated in training,” so I guess I should say “even in the absence of spurious correlations among basic features.”
… That’s exactly how I would rephrase the issue and I’m not clear on why you’re making a sharp distinction here.
Focusing again on the Monster gridworld setting, here are two different ways that your goals could misgeneralize:
1. player_has_shield is spuriously correlated with high_score during training, so the agent comes to value both
2. monster_present XOR high_score is spuriously correlated with high_score during training, so the agent comes to value both.
These are pretty different things that could go wrong. Before realizing that these crazy XOR features existed, I would only have worried about (1); now that I know these crazy XOR features exist … I think I mostly don’t need to worry about (2), but I’m not certain and it might come down to details about the setting. (Indeed, your CCS challenges work has shown that sometimes these crazy XOR features really can get in the way!)
I agree that you can think of this issue as just being the consequence of the two issues “there are lots of crazy XOR features” and “linear probes can pick up on spurious correlations,” I guess this issue feels qualitatively new to me because it just seems pretty untractable to deal with it on the data augmentation level (how do you control for spurious correlations with arbitrary boolean functions of undesired features?). I think you mostly need to hope that it doesn’t matter (because the crazy XOR directions aren’t too salient) or come up with some new idea.
I’ll note that if it ends up these XOR directions don’t matter for generalization in practice, then I start to feel better about CCS (along with other linear probing techniques).^[1]
my main claim is that it shouldn’t be surprising
If I had to articulate my reason for being surprised here, it’d be something like:
1. I didn’t expect LLMs to compute many XORs incidentally
2. I didn’t expect LLMs to compute many XORs because they are useful
but lots of XORs seem to get computed anyway. So at least one of these two mechanisms is occurring a surprising (to me) amount. If there’s a lot more incidental computation, then why? (Based on Fabian’s experiments, maybe the answer is “there’s more redundancy than I expected,” which would be interesting.) If there’s a lot more intentional computation of XORs than I expected, then why? (I’ve found the speculation that LLMs might just computing a bunch of XORs up front because they don’t know what they’ll need later interesting.) I could just update my world model to “lots of XORs exist for either reasons (1) or (2),” but I sure would be interested in knowing which of (1) or (2) it is and why.
1. ^
  I know that for CCS you’re more worried about issues around correlations with features like true_according_to_Alice, but my feeling is that we might be able to handle spurious features that are that crazy and numerous, but not spurious features as crazy and numerous as these XORs.

Sam Marks 11 Jan 2024 16:11 UTC
LW: 4 AF: 3
0
AF
in reply to: Rohin Shah’s comment on: What’s up with LLMs representing XORs of arbitrary features?
I agree with a lot of this, but some notes:
Exponentially many features
[...]
On utility explanations, you would expect that multi-way XORs are much less useful for getting low loss than two-way XORs, and so computation for multi-way XORs is never developed.
The thing that’s confusing here is that the two-way XORs that my experiments are looking at just seem clearly not useful for anything. So I think any utility explanation that’s going to be correct needs to be a somewhat subtle one of the form “the model doesn’t initially know which XORs will be useful, so it just dumbly computes way more XORs than it needs, including XORs which are never used in any example in training.” Or in other words “the model has learned the algorithm ‘compute lots of XORs’ rather than having learned specific XORs which it’s useful to compute.”
I think this subtlety changes the story a bit. One way that it changes the story is that you can’t just say “the model won’t compute multi-way XORs because they’re not useful”—the two-way XORs were already not useful! You instead need to argue that the model is implementing an algorithm which computed all the two-way XORs but didn’t compute XORs of XORs; it seems like this algorithm might need to encode somewhere information about which directions correspond to basic features and which don’t.
On the other hand, RAX introduces a qualitatively new way that linear probes can fail to learn good directions. Suppose a is a feature you care about (e.g. “true vs. false statements”) and b is some unrelated feature which is constant in your training data (e.g. b = “relates to geography”). [...]
Fwiw, failures like this seem plausible without RAX as well. We explicitly make this argument in our goal misgeneralization paper (bottom of page 9 / Section 4.2), and many of our examples follow this pattern (e.g. in Monster Gridworld, you see a distribution shift from “there is almost always a monster present” in training to “there are no monsters present” at test time).
Even though on a surface level this resembles the failure discussed in the post (because one feature is held fixed during training), I strongly expect that the sorts of failures you cite here are really generalization failure for “the usual reasons” of spurious correlations during training. For example, during training (because monsters are present), “get a high score” and “pick up shields” are correlated, so the agents learn to value picking up shields. I predict that if you modified the train set so that it’s no longer useful to pick up shields (but monsters are still present), then the agent would no longer pick up shields, and so would no longer misgeneralize in this particular way.
In contrast, the point I’m trying to make in the post is that RAX can cause problems even in the absence of spurious correlations like this.^[1]
I don’t think the model has to do any active tracking; on both hypotheses this happens by default (in incidental explanations, because of the decay postulate, and in utility explanations, because the $a \oplus b$ feature is less useful and so fewer resources go towards computing it).
As you noted, it will sometimes be the case that XOR features are more like basic features than derived features, and thus will be represented with high salience. I think incidental hypotheses will have a really hard time explaining this—do you agree?
For utility hypotheses, the point is that there needs to be something different in model internals which says “when computing these features represent the result with low salience, but when computing these features represent the result with high salience.” Maybe on your model this is something simple like the weights computing the basic features being larger than weights computing derived features? If so, that’s the tracking I’m talking about, and is a potential thread to pull on for distinguishing basic vs. derived features using model internals.
1. ^
  If you want you could rephrase this issue as ” $a$ and $a \oplus b$ are spuriously correlated in training,” so I guess I should say “even in the absence of spurious correlations among basic features.”

Sam Marks 8 Jan 2024 21:58 UTC
LW: 3 AF: 2
0
AF
in reply to: Vivek Hebbar’s comment on: What’s up with LLMs representing XORs of arbitrary features?
Thanks, you’re totally right about the equal variance thing—I had stupidly thought that the projection of $U ([0, 1]^{2})$ onto y = x would be uniform on $[- \frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}}]$ (obviously false!).
The case of a fully discrete distribution (supported in this case on four points) seems like a very special case of a something more general, where a “more typical” special case would be something like:
- if a, b are both false, then sample from $N (0, Σ)$
- if a is true and b is false, then sample from $N (μ_{a}, Σ)$
- if a is false and b is true then sample from $N (μ_{b}, Σ)$
- if a and b are true, then sample from $N (μ_{a} + μ_{b}, Σ)$
for some $μ_{a}, μ_{b} \in R^{n}$ and covariance matrix $Σ$ . In general, I don’t really expect the class-conditional distributions to be Gaussian, nor for the class-conditional covariances to be independent of the class. But I do expect something broadly like this, where the distributions are concentrated around their class-conditional means with probability falling off as you move further from the class-conditional mean (hence unimodality), and that the class-conditional variances are not too big relative to the distance between the clusters.
Given that longer explanation, does the unimodality thing still seem directionally wrong?

Sam Marks 8 Jan 2024 14:44 UTC
LW: 2 AF: 1
0
AF
in reply to: DanielVarga’s comment on: What’s up with LLMs representing XORs of arbitrary features?
Thanks, you’re correct that my definition breaks in this case. I will say that this situation is a bit pathological for two reasons:
1. The mode of a uniform distribution doesn’t coincide with its mean.
2. The variance of the multivariate uniform distribution $U ([0, 1] \times [0, 1])$ is largest along the direction $x_{1} + x_{2}$ , which is exactly the direction which we would want to represent a AND b.
I’m not sure exactly which assumptions should be imposed to avoid pathologies like this, but maybe something of the form: we are working with boolean features $f$ whose class-conditional distributions $D (\cdot | f), D (\cdot | \neg f)$ satisfy properties like
- $D (\cdot | f), D (\cdot | \neg f)$ are unimodal, and their modes coincide with their means
- The variance of $D (\cdot | f), D (\cdot | \neg f)$ along any direction is not too large relative to the difference of the means $E_{x \sim D (\cdot | f)} (x) - E_{x \sim D (\cdot | \neg f)} (x)$

Sam Marks 8 Jan 2024 14:36 UTC
LW: 2 AF: 1
0
AF
in reply to: Hoagy’s comment on: What’s up with LLMs representing XORs of arbitrary features?
Neat hypothesis! Do you have any ideas for how one would experimentally test this?

Sam Marks 7 Jan 2024 16:49 UTC
LW: 2 AF: 1
0
AF
in reply to: Vivek Hebbar’s comment on: What’s up with LLMs representing XORs of arbitrary features?
Some features which are computed from other features should probably themselves be treated as basic and thus represented with large salience.

Sam Marks 6 Jan 2024 0:28 UTC
LW: 14 AF: 8
2
AF
in reply to: Sam Marks’s comment on: What’s up with LLMs representing XORs of arbitrary features?
Using a dataset of 10,000 inputs of the form
[random LLaMA-13B generated text at temperature 0.8] [either the most likely next token or the 100th most likely next token, according to LLaMA-13B] ["true" or "false"] ["banana" or "shed"]
I’ve rerun the probing experiments. The possible labels are
- has_true: is the second last token “true” or “false”?
- has_banana: is the last token “banana” or “shed”?
- label: is the third last token the most likely or the 100th most likely?
(this weird last option is because I’m adapting a dataset from the Geometry of Truth paper about likely vs. unlikely text).
Here are the results for LLaMA-2-13B
And here are the results for the reset network
I was a bit surprised that the model did so badly on has_true, but in hindsight, considering that the activations are extracted over the last token and “true”/”false” is the penultimate token, this seems fine.
Mostly I view this as a sanity check to make sure that when the dataset is larger we don’t get the <<50% probe accuracies. I think to really dig into this more, one would need to do this with features which are not token-level and which are unambiguously linearly accessible (unlike the “label” feature here).
@ryan_greenblatt @abhatt349 @Fabien Roger

Sam Marks 5 Jan 2024 11:01 UTC
LW: 2 AF: 1
0
AF
in reply to: Gabriel Mukobi’s comment on: What’s up with LLMs representing XORs of arbitrary features?
Yes, you are correct, thanks. I’ll edit the post when I get a chance.

Sam Marks

How do we know we’re detecting truth, and not just likely statements?