These puzzles are great, thanks for making them!
Connor Kissane
Karma: 176
Amazing! We found your original library super useful for our Attention SAEs research, so thanks for making this!
Thanks for the comment! We always use the pre-ReLU feature activation, which is equal to the post-ReLU activation (given that the feature is activate), and is purely linear function of z. Edited the post for clarity.
Maybe I just missed it, but I’m not seeing this. Is the code still available?