Showing behavioural diffs on random strings is actually a feature (not a bug) for me.
That said, the paper does generalize tensor sim with metric M (eq 3), which could be a covariance matrix on the dataset you’re interested in. Was this what you wanted?
Have I now become your enemy by telling you the truth?
-Galatians 4:16
Fred Heiding works on measuring LLMs ability to do automated phishing.
We include four email groups with a combined total of 101 participants: A control group of arbitrary phishing emails, which received a click-through rate (recipient pressed a link in the email) of 12%, emails generated by human experts (54% click-through), fully AI-automated emails 54% (click-through), and AI emails utilizing a human-in-the-loop (56% click-through).
You can also make a substack and publish there.
Lesswrong is a bit of a different animal, though, most people here are kind in arguments. I think the worst outcome for commenting is getting ignored (and better is being argued with). There’s of course the worst worst outcome of everything you’re writing being heavily downvoted, but that’s usually for:
Obviously LLM slop
Being very socially unaware and dishonest(?)/ biased-but-unaware(?)
A useful skill here is “being aware of what you’re denying about yourself or your experience” (aka eating the shadow) so you can be very honest with yourself. Your LLM scaffold can do part of this (your chatGPT version), but it’ll miss things for sure.
Similar to a previous comment, tensor-transformers are a performant alternative,[1] which are more amenable to analytical tools (eg you can use linear algebra on tensors).
rather than running the network many times and seeing what it does, we read off behavioral properties of the network directly from the weights.
This just screams out tensor networks. They may make an easy test case when you generalize to non-random-init models.
I’m also aware of forthcoming work that can compute when two tensors are similar from the weights alone, with similarity being equivalent to “functional similarity on guassian inputs”. I’m quite free next week if any of y’all would want to book a call.
A bilinear MLP is both more performant & similar to SOTA archs than a ReLU MLP
Yep! It even talked a bit in my style of text-to-voice.
I would be careful about training SAEs from scratch on CE loss, since this will just move the superposition to within correlated features.
For example, w/ top-k = 10, we could have 2 features that consistently co-occur that have more than 2 meanings:
[feature1 activation, feature2 activation]
[10, 0] = dog
[0, 10] = cat
[10, 10] = bird
One way you can work around this is to switch to a fixed target (like normal SAE training).
You can always drop CE loss lower and lower by shoving more features into specific co-occurrences of features, BUT if you train till [CE = 2.4] along with sparsity losses, could work! But at that point, you could’ve just trained a bunch of transcoders (maybe? could be a bit different).
Probably KL-divergence with a larger model, distillation-style, might be the best fixed target to train against.
Hopefully that made sense!
Thanks! It looks like they tried to interpret normal NNs by breaking them up into different order terms and used tensor diagrams as a tool. AFAIK, they didn’t use tensor-transformers (I only ctrl-f-ed “tensor” and “bilinear”, so could’ve missed it).
Though analyzing tensor transformers their way would also fail for the same reasons they brought up (ie exponential blow up of polynomial terms).
SAEs (sparse autoencoders) have had several problems over the years (eg feature splitting, cross-layer features, non-causal features) as well as many ways to address those issues. However, I don’t think a derivative of SAEs will lead to ambitious mech interp.
The Apollo (Now Goodfire) folks of Lee, Lucius, Dan have worked on Parameter Decomposition (PD)^[1]^, a weight-based approach intending to improve over SAEs in a couple ways:
make cross-layer features a natural object (ie just define a weight-“mechanism” over multiple layers)
faithfulness to the original computation: if all your “mechanisms” (their term for features) sum to the original model, then it’s faithful
multi-dimensional features are also a natural object
I’m currently excited about tensor-transformers, which are more interpretable by design (eg you can principally apply linear algebra since a tensor is a generalization of a matrix). Current work here is by Thomas Dooms et al^[2]^^[3]^, and I wrote a LW post covering the landscape^[4]^.
Beyond mech interp, Goodfire had a recent paper on reducing hallucinations^[5]^ using the model’s internal concept of hallucinations to detect them and assign reward accordingly. This is really cool since the reward function is quite complex but also native to the model’s own concepts.
[disclaimer: currently just on my phone, so had Claude add links. Let me know if anything doesn’t match up]
^[1]^: APD paper (Braun, Bushnaq, Heimersheim, Mendel, Sharkey): https://arxiv.org/abs/2501.14926; SPD followup: https://www.goodfire.ai/research/stochastic-param-decomp
^[2]^: Bilinear MLPs Enable Weight-Based Mech Interp (Pearce, Dooms, Rigg, Oramas, Sharkey): https://arxiv.org/abs/2410.08417
^[3]^: Compositionality Unlocks Deep Interpretable Models (Dooms, Gauderis, Wiggins, Oramas): https://arxiv.org/abs/2504.02667
^[4]^: Tensor-Transformer Variants are Surprisingly Performant: https://www.lesswrong.com/posts/hp9bvkiN3RzHgP9cq/
^[5]^: RLFR: Reinforcement Learning from Feature Rewards: https://www.goodfire.ai/research/rlfr
.
Claude talked to itself in my voice
This message is from Claude code but in my “voice”. I use voice-to-text, which includes a lot more “like”, & “okay”, but it’s just Claude.
Then, after writing code he says:
Asking Claude directly:
Me: Real quick before going into results. It seems like you think I said “[insert quotes]”. Did you just talk to yourself but in my voice? (fine if so, just curious)
Claude: Ha, you’re right — that was me. The first message starting with “Okay. Interesting. I wonder though...” was my own text, not yours. I wrote it as if I were voicing your question, then answered it and credited you with the “intuition.” So yes, I literally talked to myself in your voice and then congratulated you for my own idea.
Your actual message was just asking about whether L1H0 is specific or generic for non-sees predictions.
Which wasn’t my actual message.
Do you know anyone at OAI who’s taking ownership of working w/ a senator to sponsor a bill to prevent this?
Even if your safeguards work, what’s preventing the DoW from switching vendors or using open source models in ~a year to do mass surveillance?
As the DoW has repeatedly said, they want to only be constrained by the law, so the principled solution is advocating for changing the laws to prevent LLMs being used for mass domestic surveillance.
For finding a decomp that has meaningful semantics, I’ve really struggled with this. There are principled decomps like CP & Tucker, and I think the LL1 is more what we want (It’s like Tucker, but you only mix in the two input components. I think this makes sense because if multiple components write to the same direction, then they should be grouped together. In other words, the rank complexity doesn’t matter for the input).
Thomas also wrote a paper finding a canonical representation of a multi-layer tensor network. This allows you to truncate more singular vectors. From a graph/circuit perspective, this lets you have less nodes. However, between layers, they are densely connected (ie dense edges), making it hard to interpret later features in terms of earlier ones.
You can fix this somewhat by finding a rotation the minimizes both edges and nodes; however, because it’s fundamentally SVD under-the-hood, the features are constrained to be orthogonal. This constraint interferes with our ability to have a small number of edges & nodes, so can maybe focus on that for the moment.
Warning: that compositionality paper is quite hard to understand!