Kay Kozaronek comments on A Barebones Guide to Mechanistic Interpretability Prerequisites

[Kay Kozaronek]

Thanks for putting this together Neel, I think you achieved your goal of making it fairly unintimidating.

One quick note: all of the links in this section are outdated. Perhaps you can update them.

Good (but hard) exercise: Code your own tiny GPT-2 and train it. If you can do this, I’d say that you basically fully understand the transformer architecture.

• Example of basic training boilerplate and train script

• The EasyTransformer codebase is probably good to riff off of here