Very much appreciate the link post—I’d been trying to write a summary/contextualisation for LW and this is a much better one than I’d come up with.
I’d be very grateful for the LW community’s thoughts (especially any pushback). I expect this will be the source of the strongest counterarguments.
Thanks for the summary! Your first bullet point was my motivation for doing this. I think it’s important to test out interpretability ideas in more challenging domains.
We didn’t really do much interpretability in this paper, this is more meta-interpretability in a sense (i.e. studying whether interpretability should in principle be possible). I’d say section 4 is worth a look, especially section 4.5 which covers fundamental and practical challenges to probing. Section 7 has some NMF analysis, and we open-sourced NMF factors which you might find interesting.
Thanks! I really like inductive vs deductive and would probably have used them if I’d thought of it.
I’m one of the authors on this paper—happy to answer any questions/discuss if anyone is interested.