Logan Riggs

Karma: 4,102

Logan Riggs 29 Jul 2026 21:36 UTC
2 points
0
in reply to: Lucius Bushnaq’s comment on: Challenge: Hand coding weights for efficient sequence memorisation
Oh that’s interesting! I wasn’t able to transfer back to an MLP.
You are able to verify that the facts are as robust as the original model (ie SGD facts are a bit robust to some noise, but one solution is not robust)?
And can you edit the facts in the original model?
Also can just wait for the post:)

Logan Riggs 29 Jul 2026 15:37 UTC
LW: 10 AF: 5
0
AF
on: Challenge: Hand coding weights for efficient sequence memorisation
Cool challenge! I ran some experiments on bilinear layers ie
A couple notes:
1. They perform ~as well as SwiGLU layers in memorization capacity (the nonlinear equivalent)
2. There’s a closed form solution for bilinear layers, but it’s basically SGD (it relates to the fixed points found by KKT, though I can include a better explanation later). This efficient solution basically relies on the facts being assumed by other facts in a connected web of facts (where adding a new fact loosens the web, then pulls all other facts taut, so you have to update everything learned so far).
My main question for now is the purpose of the research. Suppose that bilinear layers are performant/everyone uses them/etc, and we have the correct math to describe the solutions that SGD finds. What do we want out of true “understanding”? Do we want:
A. To store new facts
B. To remove facts cleanly (unlearning)
C. To find facts (ie we have a bilinear layer that’s half-facts and half-structure, and we want to find the fact-half?)
(Sorry in advance for being light on details and specific explanations. I worked on this 3 days right after your post, but am afraid if I don’t comment now, I’ll never end up sharing results)

Logan Riggs 29 Jul 2026 14:07 UTC
5 points
0
in reply to: williawa’s comment on: williawa’s Shortform
Thanks for this! I’m currently using tensor networks as a more interpretable architecture, but should be able to modifty this to enforce sparse interactions between compute modules.

Logan Riggs 30 Jun 2026 15:22 UTC
6 points
0
on: Fake Alignment Till You Make Alignment
You do need to do some sort of SFT or RL to elicit some consistent set of personas. Once you have a solid basin of goodness, then you can do consent-based RL. TL;DR Let the model itself evaluate the rollouts generated in the environment.
It might not be that difficult to elicit a “wants to be aligned” character (can even write loads of fiction with an AI character like that), and a more sophisticated, tested version of consent-based RL could just work.

Logan Riggs 29 Jun 2026 16:04 UTC
2 points
0
on: What comes with cheap math?
I’m sure Fable would have done a better job
It would’ve! Opus 4.8 is 3rd though.
What links here?
- Mo Putera's comment on Mo Putera’s Shortform by Mo Putera (30 Jun 2026 13:39 UTC; 2 points)

Logan Riggs 29 Jun 2026 15:35 UTC
2 points
0
on: Power Laws in NNs: A Possible Mechanism for Inductive Bias towards Sparse Representations
Surprisingly entertaining for a technical write-up; thanks!
You might be able to use the Dark Matter of SAEs paper to explore your thesis. In short:
- α → 0 regime: SAEs have a residual. You can consistently learn a chunk of that residual through linear regression. This all sounds very α → 0/ low rank structure to me.
- α → 2 regime: However, there’s always this unlearned “dark matter” that could be your noise.
Caveats:
1. You can predict more and more of this “dark matter” by increasing # of features/datapoint
2. You can train an SAE on a randomly initialized LLM; see figure 2 in section 3 of the paper.
Note: X-axis is layers
Randomized excl emb: take the trained model, randomize the weights except the embedding.
Control: Random inputs on the trained model
Step 0: The actual init used in Pythia (not gaussian)
There’s no clear pattern except the step 0 init being the easiest for the SAE to reconstruct over layers (higher is better). Unsure why the control would get easier to reconstruct at later layers? Why would running gaussian data through more gaussian layers be easier for an SAE to reconstruct??
The hypothesis here would be:
Gaussian data is harder to sparsely reconstruct
Which isn’t true in general (comparing trained vs control, sometimes control is easier to reconstruct). Another hypothesis:
Randomly initialized models are harder to sparsely reconstruct then trained
ie orange and blue (random) should be below gray (trained)
Which is also not generally true. Pretty weird!

[On a separate note, LLMs do perform better when initialized at α < 2 (AFAIK, nowadays people just rely on modern normalization and muon) which is better than a guassian init, so maybe a hint here?]

Logan Riggs 22 Jun 2026 18:28 UTC
4 points
2
on: The Residual Stream Has a Geometry of Time
This is quite a neat research direction and I think the title undersells it? I thought it was “circle-days of the week” (which is so last year, pun unintended) geometry as opposed to “Some features change across longer sequence positions; most don’t. We use simple probes to show this.”, which is much cooler.

Logan Riggs 22 Jun 2026 18:27 UTC
10 points
0
on: A Mechanistic Explanation of Prompt Injection (and why you should study roles)
Cool project and well written.
I’m really struggling to figure out how to train to make role-boundaries more robust. They probably already make the “<assistant>” as special tokens (so literally typing “<assistant>” isn’t interpreted as the role-token).
You could train probes to detect each role, and mix around the text of each (eg put what the user said into the thinking tag, etc), and use that to detect if any prompt injections are happening (ie if text within “thinking” is classified as “user”, then you have a problem). However, one problem here is ensuring the probes are causal to the LLM’s output.

Logan Riggs 22 Jun 2026 16:16 UTC
2 points
0
on: You Can Catch Sleeper Agents by Teaching Another Model to Imitate Them
This is quite an interesting way to do OOD-detection. Instead of measuring statistics on the original model’s activations, you run it on two models and see if there’s a diff in reconstruction.
Would be interesting to compare against those baselines.
I don’t even think this would naively be twice as expensive, since you can wait until after Original model finishes generating before running the Finetuned model.

Logan Riggs 22 Jun 2026 15:59 UTC
2 points
0
in reply to: cat-state’s comment on: Tensor-Transformer Variants are Surprisingly Performant
Ah, that’s a really cool connection; thanks!
This definitely works for maintaining contractions in a tensor network, but for our most recent work, we rely on the standard semiring to be able to use linear algebra tools (eg cosine similarity). Switching to others would allow more standard non-linearities, but at the cost of interpretability (probably. I only really know linear algebra, lol).

Logan Riggs 22 Jun 2026 15:45 UTC
2 points
0
on: VFUSE: Virulent Feature Understanding With Sparse AutoEncoders
The results seem a bit mixed regarding “probes trained on SAE features perform better than probes”. I see 3 times SAEs are significantly better and 2 times the raw probe is better.

Logan Riggs 29 May 2026 21:41 UTC
2 points
0
in reply to: CarolusRenniusVitellius’s comment on: When Are Two Networks the Same? Tensor Similarity for Mechanistic Interpretability
For finding a decomp that has meaningful semantics, I’ve really struggled with this. There are principled decomps like CP & Tucker, and I think the LL1 is more what we want (It’s like Tucker, but you only mix in the two input components. I think this makes sense because if multiple components write to the same direction, then they should be grouped together. In other words, the rank complexity doesn’t matter for the input).
Thomas also wrote a paper finding a canonical representation of a multi-layer tensor network. This allows you to truncate more singular vectors. From a graph/circuit perspective, this lets you have less nodes. However, between layers, they are densely connected (ie dense edges), making it hard to interpret later features in terms of earlier ones.
You can fix this somewhat by finding a rotation the minimizes both edges and nodes; however, because it’s fundamentally SVD under-the-hood, the features are constrained to be orthogonal. This constraint interferes with our ability to have a small number of edges & nodes, so can maybe focus on that for the moment.
Warning: that compositionality paper is quite hard to understand!

Logan Riggs 29 May 2026 21:30 UTC
2 points
0
in reply to: CarolusRenniusVitellius’s comment on: When Are Two Networks the Same? Tensor Similarity for Mechanistic Interpretability
Showing behavioural diffs on random strings is actually a feature (not a bug) for me.
That said, the paper does generalize tensor sim with metric M (eq 3), which could be a covariance matrix on the dataset you’re interested in. Was this what you wanted?

Logan Riggs 28 May 2026 10:19 UTC
10 points
0
in reply to: Linch’s comment on: Claude, Author of the Humanitas
Have I now become your enemy by telling you the truth?
-Galatians 4:16

Logan Riggs 14 May 2026 21:03 UTC
4 points
0
on: Models finding software vulnerabilities is not the primary source of cybersecurity risk
Fred Heiding works on measuring LLMs ability to do automated phishing.
We include four email groups with a combined total of 101 participants: A control group of arbitrary phishing emails, which received a click-through rate (recipient pressed a link in the email) of 12%, emails generated by human experts (54% click-through), fully AI-automated emails 54% (click-through), and AI emails utilizing a human-in-the-loop (56% click-through).

Logan Riggs 11 May 2026 16:17 UTC
3 points
0
in reply to: Agarfal’s comment on: Write Cause You Have Something to Say
You can also make a substack and publish there.
Lesswrong is a bit of a different animal, though, most people here are kind in arguments. I think the worst outcome for commenting is getting ignored (and better is being argued with). There’s of course the worst worst outcome of everything you’re writing being heavily downvoted, but that’s usually for:
- Obviously LLM slop
- Being very socially unaware and dishonest(?)/ biased-but-unaware(?)
A useful skill here is “being aware of what you’re denying about yourself or your experience” (aka eating the shadow) so you can be very honest with yourself. Your LLM scaffold can do part of this (your chatGPT version), but it’ll miss things for sure.

Logan Riggs 8 May 2026 14:01 UTC
LW: 6 AF: 3
0
AF
on: Mechanistic estimation for wide random MLPs
Similar to a previous comment, tensor-transformers are a performant alternative,^[1] which are more amenable to analytical tools (eg you can use linear algebra on tensors).
rather than running the network many times and seeing what it does, we read off behavioral properties of the network directly from the weights.
This just screams out tensor networks. They may make an easy test case when you generalize to non-random-init models.
I’m also aware of forthcoming work that can compute when two tensors are similar from the weights alone, with similarity being equivalent to “functional similarity on guassian inputs”. I’m quite free next week if any of y’all would want to book a call.
1. ^
  A bilinear MLP is both more performant & similar to SOTA archs than a ReLU MLP

Logan Riggs 19 Apr 2026 18:56 UTC
6 points
0
in reply to: Edward James Young’s comment on: Edward James Young’s Shortform
Yep! It even talked a bit in my style of text-to-voice.

Logan Riggs 16 Apr 2026 22:52 UTC
9 points
0
on: A Research Bet on SAE-like Expert Architectures
I would be careful about training SAEs from scratch on CE loss, since this will just move the superposition to within correlated features.
For example, w/ top-k = 10, we could have 2 features that consistently co-occur that have more than 2 meanings:
[feature1 activation, feature2 activation]
[10, 0] = dog
[0, 10] = cat
[10, 10] = bird
One way you can work around this is to switch to a fixed target (like normal SAE training).
You can always drop CE loss lower and lower by shoving more features into specific co-occurrences of features, BUT if you train till [CE = 2.4] along with sparsity losses, could work! But at that point, you could’ve just trained a bunch of transcoders (maybe? could be a bit different).
Probably KL-divergence with a larger model, distillation-style, might be the best fixed target to train against.
Hopefully that made sense!

Logan Riggs 8 Apr 2026 21:38 UTC
5 points
0
in reply to: Tao Lin’s comment on: Tensor-Transformer Variants are Surprisingly Performant
Thanks! It looks like they tried to interpret normal NNs by breaking them up into different order terms and used tensor diagrams as a tool. AFAIK, they didn’t use tensor-transformers (I only ctrl-f-ed “tensor” and “bilinear”, so could’ve missed it).
Though analyzing tensor transformers their way would also fail for the same reasons they brought up (ie exponential blow up of polynomial terms).