Ryan Kidd comments on Ryan Kidd’s Shortform

[Ryan Kidd]

I made a Google Scholar page for MATS. This was inspired by @Esben Kran’s Google Scholar for Apart Research. Eleuther AI subsequently made one too. I think all AI safety organizations and research programs should consider making Google Scholar pages to better share research and track impact.

[Ryan Kidd]

Here is a plot of the annual citations received by MATS, EleutherAI, and Apart research, adjusted so they start on the same year. The three organizations are somewhat comparable, as they leverage large networks of external collaborators: MATS mentors/​fellows, EleutherAI Discord, Apart sprint participants.

The EleutherAI data fits a logistic curve perfectly, asymptoting to ~18.5k citations/​year. I can’t fit the others as at least 4 data points are needed to fit a logistic curve.

[Ryan Kidd]

The top-10 most-cited papers that MATS contributed to are (all with at least 290 citations)

1. Representation Engineering: A Top-Down Approach to AI Transparency

2. Sparse autoencoders find highly interpretable features in language models

3. Towards understanding sycophancy in language models

4. Steering Language Models With Activation Engineering

5. Steering Llama 2 via Contrastive Activation Addition

6. Refusal in language models is mediated by a single direction

7. The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning

8. The Reversal Curse: LLMs trained on” A is B” fail to learn” B is A”

9. LLM Evaluators Recognize and Favor Their Own Generations

10. Finding neurons in a haystack: Case studies with sparse probing

Compare this to the top-10 highest-karma LessWrong posts that MATS contributed to (all with over 200 karma):

1. SolidGoldMagikarp (plus, prompt generation)

2. Steering GPT-2-XL by adding an activation vector (arXiv)

3. Transformers Represent Belief State Geometry in their Residual Stream (arXiv)

4. Understanding and Controlling a Maze-Solving Policy Network (arXiv)

5. Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs (arXiv)

6. Refusal in LLMs is mediated by a single direction (arXiv)

7. Natural Abstractions: Key Claims, Theorems, and Critiques

8. Distillation Robustifies Unlearning (arXiv)

9. Mechanistically Eliciting Latent Behaviors in Language Models

10. Neural networks generalize because of this one weird trick