Ryan Kidd comments on Ryan Kidd’s Shortform

[Ryan Kidd]

The top-10 most-cited papers that MATS contributed to are (all with at least 290 citations)

1. Representation Engineering: A Top-Down Approach to AI Transparency

2. Sparse autoencoders find highly interpretable features in language models

3. Towards understanding sycophancy in language models

4. Steering Language Models With Activation Engineering

5. Steering Llama 2 via Contrastive Activation Addition

6. Refusal in language models is mediated by a single direction

7. The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning

8. The Reversal Curse: LLMs trained on” A is B” fail to learn” B is A”

9. LLM Evaluators Recognize and Favor Their Own Generations

10. Finding neurons in a haystack: Case studies with sparse probing

Compare this to the top-10 highest-karma LessWrong posts that MATS contributed to (all with over 200 karma):

1. SolidGoldMagikarp (plus, prompt generation)

2. Steering GPT-2-XL by adding an activation vector (arXiv)

3. Transformers Represent Belief State Geometry in their Residual Stream (arXiv)

4. Understanding and Controlling a Maze-Solving Policy Network (arXiv)

5. Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs (arXiv)

6. Refusal in LLMs is mediated by a single direction (arXiv)

7. Natural Abstractions: Key Claims, Theorems, and Critiques

8. Distillation Robustifies Unlearning (arXiv)

9. Mechanistically Eliciting Latent Behaviors in Language Models

10. Neural networks generalize because of this one weird trick