Progress Report 5: tying it together

Previous: Progress report 4

First, an example of what not to do if you want humanity to survive: make an even foom-ier and less interpretable version of neural nets. On the spectrum of good idea to bad idea, this one is way worse than neuromorphic computing. In fact, they even had a paragraph in their paper discussing their method in contrast to Hebbian learning, and showing how their method is more volatile and unpredictable. Great. https://​​​​abs/​​2202.05780

On the flip side of the coin, here’s some good stuff that I’m happy to se being worked on.

Some other work that has been done with nostalgebraist’s logit-lens: Looking for Grammar in all the Right Places Alethea Power

Tom Frederik working on a projected called ‘Unseal’ https://​​TomFrederik/​unseal (to unseal the mystery of transformers), expanding on the PySvelte library to include aspects of the logit-lens idea.

Inspired by this, I started my own fork from his library to work on a interactive visualization that creates an open-ended framework for tying together multiple ways of interpreting models. I’m hoping to use it to put together the stuff I’ve found or made so far, as well as whatever I find or make next. It just seems generally useful to have some sort of way to put a bunch of different ‘views’ of the model together side-by-side in order to scan across all of them to get a more inclusive view of things.

Once I get this tool a bit more fleshed out, I plan to start trying to plan the audit game with it, using toy models that have been edited.

Here’s a presentation of my ideas I made for the conclusion of the AGI safety fundamentals course. https://​​​​FarRgPwBpGU

Among the things I’m thinking of putting in are ideas related to these papers.

One: A paper mentioned by jsteinhardt here https://​​​​posts/​​qAhT2qvKXboXqLk4e/​​early-2022-paper-round-up : Summarizing Differences between Text Distributions with Natural Language (w/​ Ruiqi Zhong, Charlie Snell, Dan Klein)

This paper discusses their language data summarization technique, which I think will be cool to use someday, but also along the way to building that they had to do some clustering which I think sounds like a useful thing to include in my visualization and contrast with the neuron-importance topic clustering. I hope to also revisit and improve on the neuron-importance clustering. I think if I repeat the neuron-importance sampling on topic-labeled samples, I’ll then be able to tag the resulting clusters with the topics they most strongly relate to. That will make them more useful for interpretation.

Two: https://​​​​abs/​​2203.14680 Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space Mor Geva, Avi Caciularu, Kevin Ro Wang, Yoav Goldberg


What does BERT dream of? Deep dream with text https://​​​​docs/​​www/​​​​c331351a690011a2a37f7ee1c75bf771f01df3a3.html

Seems neat and sorta related. I can probably figure out some way to add a version of this text-deep-dreaming to the laundry list of ‘windows into interpretability’ I’m accumulating.

No comments.