Oh that’s interesting! I wasn’t able to transfer back to an MLP.
You are able to verify that the facts are as robust as the original model (ie SGD facts are a bit robust to some noise, but one solution is not robust)?
And can you edit the facts in the original model?
Also can just wait for the post:)
Cool challenge! I ran some experiments on bilinear layers ie
A couple notes:
They perform ~as well as SwiGLU layers in memorization capacity (the nonlinear equivalent)
There’s a closed form solution for bilinear layers, but it’s basically SGD (it relates to the fixed points found by KKT, though I can include a better explanation later). This efficient solution basically relies on the facts being assumed by other facts in a connected web of facts (where adding a new fact loosens the web, then pulls all other facts taut, so you have to update everything learned so far).
My main question for now is the purpose of the research. Suppose that bilinear layers are performant/everyone uses them/etc, and we have the correct math to describe the solutions that SGD finds. What do we want out of true “understanding”? Do we want:
A. To store new facts
B. To remove facts cleanly (unlearning)
C. To find facts (ie we have a bilinear layer that’s half-facts and half-structure, and we want to find the fact-half?)
(Sorry in advance for being light on details and specific explanations. I worked on this 3 days right after your post, but am afraid if I don’t comment now, I’ll never end up sharing results)
Thanks for this! I’m currently using tensor networks as a more interpretable architecture, but should be able to modifty this to enforce sparse interactions between compute modules.
You do need to do some sort of SFT or RL to elicit some consistent set of personas. Once you have a solid basin of goodness, then you can do consent-based RL. TL;DR Let the model itself evaluate the rollouts generated in the environment.
It might not be that difficult to elicit a “wants to be aligned” character (can even write loads of fiction with an AI character like that), and a more sophisticated, tested version of consent-based RL could just work.
I’m sure Fable would have done a better job
It would’ve! Opus 4.8 is 3rd though.
Surprisingly entertaining for a technical write-up; thanks!
You might be able to use the Dark Matter of SAEs paper to explore your thesis. In short:
α → 0 regime: SAEs have a residual. You can consistently learn a chunk of that residual through linear regression. This all sounds very α → 0/ low rank structure to me.
α → 2 regime: However, there’s always this unlearned “dark matter” that could be your noise.
Caveats:
1. You can predict more and more of this “dark matter” by increasing # of features/datapoint
2. You can train an SAE on a randomly initialized LLM; see figure 2 in section 3 of the paper.
Note: X-axis is layers
Randomized excl emb: take the trained model, randomize the weights except the embedding.
Control: Random inputs on the trained model
Step 0: The actual init used in Pythia (not gaussian)
There’s no clear pattern except the step 0 init being the easiest for the SAE to reconstruct over layers (higher is better). Unsure why the control would get easier to reconstruct at later layers? Why would running gaussian data through more gaussian layers be easier for an SAE to reconstruct??
The hypothesis here would be:
Gaussian data is harder to sparsely reconstruct
Which isn’t true in general (comparing trained vs control, sometimes control is easier to reconstruct). Another hypothesis:
Randomly initialized models are harder to sparsely reconstruct then trained
ie orange and blue (random) should be below gray (trained)
Which is also not generally true. Pretty weird!
[On a separate note, LLMs do perform better when initialized at α < 2 (AFAIK, nowadays people just rely on modern normalization and muon) which is better than a guassian init, so maybe a hint here?]
This is quite a neat research direction and I think the title undersells it? I thought it was “circle-days of the week” (which is so last year, pun unintended) geometry as opposed to “Some features change across longer sequence positions; most don’t. We use simple probes to show this.”, which is much cooler.
Cool project and well written.
I’m really struggling to figure out how to train to make role-boundaries more robust. They probably already make the “<assistant>” as special tokens (so literally typing “<assistant>” isn’t interpreted as the role-token).
You could train probes to detect each role, and mix around the text of each (eg put what the user said into the thinking tag, etc), and use that to detect if any prompt injections are happening (ie if text within “thinking” is classified as “user”, then you have a problem). However, one problem here is ensuring the probes are causal to the LLM’s output.
This is quite an interesting way to do OOD-detection. Instead of measuring statistics on the original model’s activations, you run it on two models and see if there’s a diff in reconstruction.
Would be interesting to compare against those baselines.
I don’t even think this would naively be twice as expensive, since you can wait until after Original model finishes generating before running the Finetuned model.
Ah, that’s a really cool connection; thanks!
This definitely works for maintaining contractions in a tensor network, but for our most recent work, we rely on the standard semiring to be able to use linear algebra tools (eg cosine similarity). Switching to others would allow more standard non-linearities, but at the cost of interpretability (probably. I only really know linear algebra, lol).
The results seem a bit mixed regarding “probes trained on SAE features perform better than probes”. I see 3 times SAEs are significantly better and 2 times the raw probe is better.
For finding a decomp that has meaningful semantics, I’ve really struggled with this. There are principled decomps like CP & Tucker, and I think the LL1 is more what we want (It’s like Tucker, but you only mix in the two input components. I think this makes sense because if multiple components write to the same direction, then they should be grouped together. In other words, the rank complexity doesn’t matter for the input).
Thomas also wrote a paper finding a canonical representation of a multi-layer tensor network. This allows you to truncate more singular vectors. From a graph/circuit perspective, this lets you have less nodes. However, between layers, they are densely connected (ie dense edges), making it hard to interpret later features in terms of earlier ones.
You can fix this somewhat by finding a rotation the minimizes both edges and nodes; however, because it’s fundamentally SVD under-the-hood, the features are constrained to be orthogonal. This constraint interferes with our ability to have a small number of edges & nodes, so can maybe focus on that for the moment.
Warning: that compositionality paper is quite hard to understand!
Showing behavioural diffs on random strings is actually a feature (not a bug) for me.
That said, the paper does generalize tensor sim with metric M (eq 3), which could be a covariance matrix on the dataset you’re interested in. Was this what you wanted?
Have I now become your enemy by telling you the truth?
-Galatians 4:16
Fred Heiding works on measuring LLMs ability to do automated phishing.
We include four email groups with a combined total of 101 participants: A control group of arbitrary phishing emails, which received a click-through rate (recipient pressed a link in the email) of 12%, emails generated by human experts (54% click-through), fully AI-automated emails 54% (click-through), and AI emails utilizing a human-in-the-loop (56% click-through).
You can also make a substack and publish there.
Lesswrong is a bit of a different animal, though, most people here are kind in arguments. I think the worst outcome for commenting is getting ignored (and better is being argued with). There’s of course the worst worst outcome of everything you’re writing being heavily downvoted, but that’s usually for:
Obviously LLM slop
Being very socially unaware and dishonest(?)/ biased-but-unaware(?)
A useful skill here is “being aware of what you’re denying about yourself or your experience” (aka eating the shadow) so you can be very honest with yourself. Your LLM scaffold can do part of this (your chatGPT version), but it’ll miss things for sure.
Speculating the difference between those who “feel the AGI” vs don’t:
People are social: If most people they see aren’t acting impressed, then they shouldn’t either. A vibe I get from social media is that AI is bad (for environment, jobs, AND capabilities). This is also similar to reactions to COVID early 2020. This is also similar to myself where I didn’t invest time into AI safety until I made 1 friend to work on it over discord (it became socially viable to do at that point).
AGI soon would be inconvenient to our own narrative: I remember spending a few hours to understand the chinchilla scaling laws; my stomach sunk. In the back of my mind, I had always expected to, you know, get married, have kids, live a long and healthy life, and this punctured a hole in that.[1] I was quite depressed for a couple of days, but fortunately had read how to greave well.
For students, a large narrative is finishing their degree & getting a job. Even if timelines aren’t that short, does it really make sense for them to keep studying that when a (metaphorical) meteor could hit earth in a few years? I think it’s a natural learned behavior to ignore thoughts/feelings of dissent in your current narrative you’re relying on, because no narrative --> no motivation (and it might look bad to acknowledge faults in your plan).[2]
A few related ones:
AI hasn’t literally cured cancer (I do believe there will be a cascading effect once it has done something of this caliber)
Scaling laws aren’t felt in a gut sense: I learned it in a way that viscerally affected my predictions of the future. Most people DON’T have a strong model of scaling laws. Although, I think some equations or graphs would lack punch without going into depth the capabilities of eg GPT-2,3,4, etc.
I mean, I am married now and plan to have kids, but this update was ~4 years ago.
I view this as a local optimum, cause paying more attention to dissenting stuff requires skill to be done right.