Noa Nabeshima

Karma: 210

Noa Nabeshima 24 May 2025 21:24 UTC
1 point
0
on: Will Jesus Christ return in an election year?

The Incorrect Resolution hypothesis rings false because, while there’s some chance of an incorrect resolution, it’s really unlikely to be as high as 3%. Polymarket has a lot of reputation to lose by incorrectly resolving this market, and would almost certainly override its consensus-based resolution mechanism if it came to that.

I think historically UMA oracle token holder profiteers have misresolved some markets without Polymarket overriding.

I would guess profiteers would focus on markets with more liquidity. I would guess that the base rate so far has been much less than 3%, but maybe there are lots of resolved markets with lower trading volume (‘Will Jesus Christ...’ has $400K in volume). It does seem like it would be unusually bad for Polymarket if this particular market was misresolved.

It seems not impossible to me for many markets to be misresolved by UMA in one big attack, but I don’t know how UMA and Polymarket work.

Noa Nabeshima 7 May 2025 0:50 UTC
16 points
8
on: Noa Nabeshima’s Shortform
While language models plausibly are trained with comparable amounts of FLOP to humans today here are some differences:
- Humans process much less data
- Humans spend much more compute per datapoint
- Human data includes them taking actions and the results of those actions, language model pretraining data much less so.
These might explain some of the strengths/weaknesses of language models
- LMs know many more things than humans, but often in shallower ways.
- LMs seem less sample-efficient than humans (less compute per datapoint and they haven’t been very optimized for sample-efficiency yet)
- LMs are worse at taking actions over time than humans.

Noa Nabeshima 29 Mar 2025 22:42 UTC
5 points
1
in reply to: Noa Nabeshima’s comment on: Efficient Dictionary Learning with Switch Sparse Autoencoders
If this is your implementation:

https://github.com/amudide/switch_sae/blob/main/dictionary_learning/trainers/switch.py

It looks like you when you encode you do a dense encoder forward pass and then mask using the expert router.

I think this means that the FLOP scaling laws claim is misleading because (my impression is that) your current train code uses much more FLOP than the scaling law graphs, because it calculates every expert’s activations for every input.

But I think the empirical claims about the learned features and the FLOP scaling laws still should hold up for implementations that actually do the conditional computations.

I also expect H100/B100-time scaling charts than FLOP charts to be more informative for future work because I now think memory-bandwidth has decent odds of being the main bottleneck for training time.

Noa Nabeshima 15 Jan 2025 21:32 UTC
1 point
0
on: Noa Nabeshima’s Shortform
One barrier to SAE circuits is that it’s currently hard to understand how attention out SAE latents are calculated. Even if you do IG attribution patching to try to understand which earlier latents are relevant to the attention out SAE latents, it doesn’t tell you how these latents interact inside the attention layer at all.

Noa Nabeshima 15 Jan 2025 21:27 UTC
2 points
0
on: Noa Nabeshima’s Shortform
Auto-interp is currently really really bad

I think o1 is the only model that seems to perform decently at auto-interp but it’s very expensive! IE $1/latent label. This is frustrating to me.

Noa Nabeshima 15 Jan 2025 21:18 UTC
3 points
0
on: Noa Nabeshima’s Shortform
TinyModel SAEs have these first entity and second entity latents.

E.g. if the story is ‘Once upon a time Tim met his friend Sally.’, Tim is the first entity and Sally is the second entity. The latents fire on all instances of first|second entity after the first introduction of that entity.

I think I at one point found an ‘object owned by second entity’ latent but have had trouble finding it again.

I wonder if LMs are generating these reusable ‘pointers’ and then doing computation with the pointers. For example to track that an object is owned by the first entity, you just need to calculate which entities are instances of the first entity, calculate when first entity is shown to own an object and write ‘owned by first entity’ to the object token, and then broadcast that forward to other instances of the object. Then, if you have the tokens Tim|'s

(and 's has calculated that the first entity is immediately before it), 's can, with a single attention head, look for objects owned by the first entity.

This means that the exact identity information of the object (e.g. ′ hammer’) and the exact identity information of the first entity (′ Tim’) don’t need to be passed around in computations, you can just do much cheaper pointer calculations and grab the relevant identity information when necessary.

This suggests a more fine-grained story for what duplicate name heads are doing in IOI.

Noa Nabeshima’s Shortform

Noa Nabeshima15 Jan 2025 21:18 UTC

3 points

4 comments LW link

Noa Nabeshima 30 Dec 2024 23:48 UTC
LW: 2 AF: 2
0
AF
in reply to: chanind’s comment on: Matryoshka Sparse Autoencoders
You know I was thinking ab this—say that there are two children and they’re orthogonal to the parent and each have probability 0.4 given the parent. If you imagine the space it looks like three clusters, two with probability 0.4, norm 1.4 and one with probability 0.2 and norm 1. They all have high cosine similarity with each other. From this frame, having the parent ‘include’ the children directions a bit doesn’t seem that inappropriate. One SAE latent setup that seems pretty reasonable is to actually have one parent latent that’s like “one of these three clusters is active” and three child latents pointing to each of the three clusters. The parent latent decoder in that setup would also include a bit of the child feature directions.

This is all sketchy though. It doesn’t feel like we have a good answer to the question “How exactly do we want the SAEs to behave in various scenarios?”

Noa Nabeshima 28 Dec 2024 7:43 UTC
LW: 3 AF: 3
0
AF
in reply to: chanind’s comment on: Matryoshka Sparse Autoencoders
This is cool! I wonder if it can be fixed. I imagine it could be improved some amount by nudging the prefix distribution, but it doesn’t seem like that will solve it properly. Curious if this is a large issue in real LMs. It’s frustrating that there aren’t ground-truth features we have access to in language models.

I think how large of a problem this is can probably be inferred from a description of the feature distribution. It’d be nice to have a better sense of what that distribution is (assuming the paradigm is correct enough).

Noa Nabeshima 20 Dec 2024 23:41 UTC
LW: 3 AF: 3
2
AF
on: Matryoshka Sparse Autoencoders
Tree Methodology
To generate the trees in sparselatents.com/tree_view, I use a variant of Masked Cosine Similarity (MCS), a metric introduced in Towards Monosemanticity. The original MCS is calculated like this: For any two latents A and B, first compute the cosine similarity between their activations, but only considering tokens where latent A is active. Then compute the same similarity, but only for tokens where latent B is active. The final MCS value is the larger of these two similarities.
Instead of taking the max, I do a directed MCS where I just consider the cosine similarity between A and B’s activations on tokens where B is active. Then, I multiply this directed MCS score by max(B activations)/max(A activations) to ignore latents that don’t fire very much. I’m not sure that this multiplication step is necessary.
I also use a higher threshold of 0.6.
Starting from a parent latent, say S/1/12, I find all latents in a larger-width SAE (say S/2) that pass the directed MCS threshold. Then, I re-apply the method to those S/2 latents to find children in S/3.
The result is often a non-tree DAG as some of the identified latents in S/3 have more than one parent in S/2. To simplify rendering, I assign these latents to the parent they have the highest score with. This obscures the true structure, but I wasn’t sure of a clean way to automatically render these DAGs.
The trees should be thought of not as fully displaying the structure of the model, but instead of surfacing small sets of latents that I expect demonstrate feature absorption when viewed together.

Noa Nabeshima 20 Dec 2024 23:12 UTC
LW: 1 AF: 1
0
AF
on: Matryoshka Sparse Autoencoders
The Matryoshka SAE trained on the toy model learn the true features on most runs, not all of them. Sometimes a small number of latents, modally one, seem to get stuck in a bad state.

Noa Nabeshima 20 Dec 2024 21:08 UTC
LW: 3 AF: 3
2
AF
on: Matryoshka Sparse Autoencoders
I thought I had some informal evidence that permuting the latents was good and after double checking some evidence I don’t feel confident that it is good.

Training without permutation seems to attain slightly better FVU/L0, has reasonable looking features at a quick glance, seems to solve the toy model at comparable rates to permuted, and is simpler to code.

Noa Nabeshima 20 Dec 2024 3:43 UTC
LW: 3 AF: 2
0
AF
on: Learning Multi-Level Features with Matryoshka SAEs
This is great work! I like that you tested on large models and your very comprehensive benchmarking. I also like the BatchTopK architecture.
It’s interesting to me that MSE has a smaller hit than cross-entropy.
Here are some notes I made:
We suspect that using a fixed group size leads to more stable training and faster convergence.
This seems plausible to me!
Should the smallest sub-SAE get gradients from all losses, or should the losses from larger sub-SAEs be stopped?
When I tried stopping the gradient from flowing from large sub-SAE losses to small it made later latents much less interpretable. I tried an approach where early latents got less gradient from larger sub-SAE losses and it seemed to also have less interpretable late latents. I don’t know what’s going on with this.
What is the effect of latent sorting that Noa uses on the benchmarks?
I tried not ordering the latents and it did comparably on FVU/L0. I vaguely recall that for mean-max correlation, permuting did worse on early latents and better on the medium latents. At a quick glance I weakly preferred the permuted SAE latents but it was very preliminary and I’m not confident in this.
I’d love to chat more with the authors, I think it’d be fun to explore our beliefs and process over the course of making the papers and compare notes and ideas.

Noa Nabeshima 16 Dec 2024 9:39 UTC
LW: 5 AF: 3
0
AF
in reply to: chanind’s comment on: Matryoshka Sparse Autoencoders
Even with all possible prefixes included in every batch the toy model learns the same small mixing between parent and children (this was best out of 2, for the first run the matryoshka didn’t represent one of the features): https://sparselatents.com/matryoshka_toy_all_prefixes.png

Here’s a hypothesis that could explain most of this mixing. If the hypothesis is true, then even if every possible prefix is included in every batch, there will still be mixing.

Hypothesis:
Regardless of the number of prefixes, there will be some prefix loss terms where
1. a parent and child feature are active
2. the parent latent is included in the prefix
3. the child latent isn’t included in the prefix.

The MSE loss in these prefix loss terms is pretty large because the child feature isn’t represented at all. This nudges the parent to slightly represent all of its children a bit.

To compensate for this, if a child feature is active and the child latent is included the prefix, it undoes the parent decoder vector’s contribution to the features of the parent’s other children.

This could explain these weird properties of the heatmap:
- Parent decoder vector has small positive cosine similarity with child features
- Child decoder vectors have small negative cosine similarity with other child features

Still unexplained by this hypothesis:
- Child decoder vectors have very small negative cosine similarity with the parent feature.

Noa Nabeshima 16 Dec 2024 3:53 UTC
LW: 3 AF: 1
0
AF
in reply to: Bart Bussmann’s comment on: Matryoshka Sparse Autoencoders
That’s very cool, I’m looking forward to seeing those results! The Top-K extension is particularly interesting, as that was something I wasn’t sure how to approach.

I imagine you’ve explored important directions I haven’t touched like better benchmarking, top-k implementation, and testing on larger models. Having multiple independent validations of an approach also seems valuable.

I’d be interested in continuing this line of research, especially circuits with Matryoshka SAEs. I’d love to hear about what directions you’re thinking of. Would you want to have a call sometime about collaboration or coordination? (I’ll DM you!)

Really looking forward to reading your post!

Noa Nabeshima 14 Dec 2024 6:34 UTC
LW: 3 AF: 3
0
AF
in reply to: leogao’s comment on: Matryoshka Sparse Autoencoders
Yes, follow up work with bigger LMs seems good!

I use number of prefix-losses per batch = 10 here; I tried 100 prefixes per batch and the learned latents looked similar at a quick glance, so I wonder if naively training with block size = 1 might not be qualitatively different. I’m not that sure and training faster with kernels on its own seems good also!

Maybe if you had a kernel for training with block size = 1 it would create surface area for figuring out how to work on absorption when latents are right next to each other in the matryoshka latent ordering.

Matryoshka Sparse Autoencoders

Noa Nabeshima14 Dec 2024 2:52 UTC

98 points

15 comments11 min readLW link

Noa Nabeshima 3 Dec 2024 21:37 UTC
5 points
0
on: Book a Time to Chat about Interp Research
I really enjoy chatting with Logan about interpretability research.

Noa Nabeshima 23 Aug 2024 20:32 UTC
2 points
0
in reply to: Noa Nabeshima’s comment on: Parasites (not a metaphor)
It’d be funny if stomach gendlin focusing content was at least partially related to worms or other random physical stuff

Noa Nabeshima 23 Aug 2024 20:29 UTC
1 point
0
on: Parasites (not a metaphor)
I tried this a week ago and I am constipated for what I think might be the first time in years but am not sure. I also think I might have less unpleasant stomach sensation overall.

Noa Nabeshima

Noa Nabeshima’s Shortform

Tree Methodology

Ma­tryoshka Sparse Autoencoders

Matryoshka Sparse Autoencoders