Surprisingly entertaining for a technical write-up; thanks!
You might be able to use the Dark Matter of SAEs paper to explore your thesis. In short:
α → 0 regime: SAEs have a residual. You can consistently learn a chunk of that residual through linear regression. This all sounds very α → 0/ low rank structure to me.
α → 2 regime: However, there’s always this unlearned “dark matter” that could be your noise.
Caveats:
1. You can predict more and more of this “dark matter” by increasing # of features/datapoint
2. You can train an SAE on a randomly initialized LLM; see figure 2 in section 3 of the paper.
Note: X-axis is layers
Randomized excl emb: take the trained model, randomize the weights except the embedding. Control: Random inputs on the trained model Step 0: The actual init used in Pythia (not gaussian)
There’s no clear pattern except the step 0 init being the easiest for the SAE to reconstruct over layers (higher is better). Unsure why the control would get easier to reconstruct at later layers? Why would running gaussian data through more gaussian layers be easier for an SAE to reconstruct??
The hypothesis here would be:
Gaussian data is harder to sparsely reconstruct
Which isn’t true in general (comparing trained vs control, sometimes control is easier to reconstruct). Another hypothesis:
Randomly initialized models are harder to sparsely reconstruct then trained ie orange and blue (random) should be below gray (trained)
Which is also not generally true. Pretty weird!
[On a separate note, LLMs do perform better when initialized at α < 2 (AFAIK, nowadays people just rely on modern normalization and muon) which is better than a guassian init, so maybe a hint here?]
Surprisingly entertaining for a technical write-up; thanks!
You might be able to use the Dark Matter of SAEs paper to explore your thesis. In short:
α → 0 regime: SAEs have a residual. You can consistently learn a chunk of that residual through linear regression. This all sounds very α → 0/ low rank structure to me.
α → 2 regime: However, there’s always this unlearned “dark matter” that could be your noise.
Caveats:
1. You can predict more and more of this “dark matter” by increasing # of features/datapoint
2. You can train an SAE on a randomly initialized LLM; see figure 2 in section 3 of the paper.
Note: X-axis is layers
Randomized excl emb: take the trained model, randomize the weights except the embedding.
Control: Random inputs on the trained model
Step 0: The actual init used in Pythia (not gaussian)
There’s no clear pattern except the step 0 init being the easiest for the SAE to reconstruct over layers (higher is better). Unsure why the control would get easier to reconstruct at later layers? Why would running gaussian data through more gaussian layers be easier for an SAE to reconstruct??
The hypothesis here would be:
Which isn’t true in general (comparing trained vs control, sometimes control is easier to reconstruct). Another hypothesis:
Which is also not generally true. Pretty weird!
[On a separate note, LLMs do perform better when initialized at α < 2 (AFAIK, nowadays people just rely on modern normalization and muon) which is better than a guassian init, so maybe a hint here?]