Awesome work! I’d be quite interested to know whether the benefits from this technique are equivalently significant with a larger SAE and also what the original perplexity was (when looking at the summary statistics table). I’ll probably reimplement at some point.
Also, kudos on the visualizations. Really love the color scales!
Awesome work! I’d be quite interested to know whether the benefits from this technique are equivalently significant with a larger SAE and also what the original perplexity was (when looking at the summary statistics table). I’ll probably reimplement at some point.
Also, kudos on the visualizations. Really love the color scales!
The original perplexity of the LLM was ~38 on the open web text slice I used. Thanks for the compliments!