Question about the “rules of the game” you present. Are you allowed to simply look at layer 0 transcoder features for the final 10 tokens—you could probably roughly estimate the input string from these features’ top activators. From you case study, it seems that you effectively look at layer 0 transcoder features for a few of the final tokens through a backwards search, but wonder if you can skip the search and simply look at transcoder features. Thank you.
RGRGRG
Karma: 61
To confirm—the weights you share, such as 0.26 and 0.23 are each individual entries in the W matrix for:
y=Wx ?
This is a casual thought and by no means something I’ve thought hard about—I’m curious whether b is a lagging indicator, which is to say, there’s actually more magic going on in the weights and once weights go through this change, b catches up to it.
Another speculative thought, let’s say we are moving from 4* → 5* and |W_3| is the new W that is taking on high magnitude. Does this occur because somehow W_3 has enough internal individual weights to jointly look at it’s two (new) neighbors’ W_i`s roughly equally?
Does the cos similarity and/or dot product of this new W_3 with its neighbors grow during the 4* → 5* transition (and does this occur prior to the change in b?)
Question about the gif—to me it looks like the phase transition is more like:
4++- to unstable 5+- to 4+- to 5-
(Unstable 5+- seems to have similar loss to 4+-).Why do we not count the large red bar as a “-” ?
Do you expect similar results (besides the fact that it would take longer to train / cost more) without using LoRA?
If I were to be accepted for this cycle, would I be expected to attend any events in Europe? To be clear, I could attend all events in and around Berkeley.
What city/country is PIBBSS based out of / where will the retreats be? (Asking as a Bay Area American without a valid passport).
For any potential funders reading this: I’d be open to starting an interpretability lab and would love to chat. I’ve been full-time on MI for about 4 months—here is some of my work: https://www.lesswrong.com/posts/vGCWzxP8ccAfqsrS3/thoughts-about-the-mechanistic-interpretability-challenge-2
I have a few PhD friends who are working for software jobs they don’t like and would be interested in joining me for a year or longer if there were funding in place (even for just the trial period Marius proposes).
My very quick take is that interpretability has yet to understand small language models and this is a valuable direction to focus on next. (more details here: https://www.lesswrong.com/posts/ESaTDKcvGdDPT57RW/seeking-feedback-on-my-mechanistic-interpretability-research )
For any potential cofounders reading this, I have applied to a few incubators and VC funds, without any success. I think some applications would be improved if I had a co-founder. If you are potentially interested in cofounding an interpretability startup and you live in the Bay Area, I’d love to meet for coffee and see if we have a shared vision and potentially apply to some of these incubators together.
I really like your ambitious MI section and I think you hit on a few interesting questions I’ve come across elsewhere:
Two researchers interpreted a 1-layer transformer network and then I interpreted it differently—there isn’t a great way to compare our explanations (or really know how similar vs different our explanations are).
With papers like the Hydra effect that demonstrate similar knowledge can be spread throughout a network, it’s not clear to if we want to/how to analyze impact—can/should we jointly ablate multiple units across different heads at once?
I’m personally unsure how to split my time between interpreting small networks vs larger ones. Should I focus 100% on interpreting 1-2 layer TinyStories LMs or is looking into 16+ layer LLMs valuable at this time?
Most weekdays, I set the goal of myself of doing twelve focused blocks of 24 minutes of object level work (my variant on Pomodoro). Once I complete these blocks, I can do whatever I want—whether it be stop working for the rest of the day, more object level work, meta work, or anything else.
If you try something like this, I’d recommend setting a goal of doing 6(?) such blocks and then letting yourself do as much or as little meta as you want; and then potentially gradually working up to 10-13 blocks.
Over the last 3 months, I’ve spent some time thinking about mech interp as a for profit service. I’ve pitched to one VC firm, interviewed for a few incubators/accelerators including ycombinator, sent out some pitch documents, co-founder dated a few potential cofounders, and chatted with potential users and some AI founders).
There are a few issues:
First, as you mention, I’m not sure if mech interp is yet ready to understand models. I recently interpreted a 1-layer model trained on a binary classification function https://www.lesswrong.com/posts/vGCWzxP8ccAfqsrS3/thoughts-about-the-mechanistic-interpretability-challenge-2 and am currently working on understanding a 1-layer language model (TinyStories-1Layer-21M). TinyStories is (much?) harder than the binary classification network (which took 24 focused days of solo research). This isn’t to say I or someone else won’t have an idea how 1 layer models work a few months from now. Once this happens, we might want to interpret multi-layer models before being ready to interpret models that are running in production.
Second, outsiders can observe that mech interp might not be far enough along to build a product around. The feedback I received from the VC firm and YC was that my ideas weren’t far enough along.
Third, I personally have not yet been able to find someone I’m excited to be cofounders with. Some people have different visions in terms of safety (some people just don’t care at all). Other people who I share a vision with, I don’t match with for other reasons.
Fourth, I’m not certain that I’ve yet found that ideal first customer—some people seem to think it’s nice to have, but frequently with language models, if you get a bad output, you can just run it again (keeping a human in the loop). To be clear, I haven’t given up on finding that ideal customer, and it could be something like government or that customer might not exist until AI models do something really bad.
Fifth, I’m unsure if I actually want to run a company. I love doing interp research and think I am quite good at it (among other things, having a software background, a PhD in Robotics, and solving puzzles). I consider myself a 10x+ engineer. At least right now, it seems like I can add more value by doing independent research rather than running a company.
For me, the first issue is the main one. Once interp is farther along, I’m open to put more time into thinking about the other issues. If anyone reading this is potentially interested in chatting, feel free to DM me.
Thank you! I’m still surprised how little most heads in L0 + L1 seem to be using the positional embeddings. L1H4 looks reasonably uniform so I could accept that maybe that somehow feeds into L2H2.
nit: do you mean 6x6 Boolean patterns not 4x4?
This is a surprising and fascinating result. Do you have attention plots of all 144 heads you could share?
I’m particularly interested in the patterns for all heads on layers 0 and 1 matching the following caption(Left: a 50x50 submatrix of LXHY’s attention pattern on a prompt from openwebtext-10k. Right: the same submatrix of LXHY’s attention pattern, when positional embeddings are averaged as described above.)
As one specific example—has RLHF, which the below post suggests was potentially was initially intended for safety, been a net negative for AI safety?
My primary safety concern is what happens if one of these analyses somehow leads to a large improvement over the state of the art. I don’t know what form this would take and it might be unexpected given the Bitter Lesson you cite above, but if it happens, what do we do then? Given this is hypothetical and the next large improvement in LMs could come elsewhere, I’m not suggesting we stop sharing now. But I think we should be prepared that there might be a point in time where we need to acknowledge such sharing leads to significantly stronger models and thus should re-evaluate sharing such eval work.
The differences between these two projects seem like an interesting case study in MI. I’ll probably refer to this a lot in the future.
Excited to see case studies comparing and contrasting our works. Not that you need my permission, but feel free to refer to this post (and if it’s interesting, this comment) as much or as little as desired.
One thing that I don’t think came out in my post is that my initial reaction to the previous solution was that it was missing some things and might even have been mostly wrong. (I’m still not certain that it’s not at least partially wrong, but this is harder to defend and I suspect might be a minority opinion).
Contrast this to your first interp challenge—I had a hypothesis of “slightly slant-y (top left to bottom right)” images for one of the classes. After reading the first paragraph of the tl;dr of their written solution to the first challenge - I was extremely confident they were correct.
One thought I’ve had, inspired by discussion (explained more later), is whether:
“label[ing] points by interpolating” is not the opposite of “developing an interesting, coherent internal algorithm.” (This is based on a quote from Stephen Casper’s retrospective that I also quoted in my post).
It could be the case that the network might have “develop[ed] an interesting, coherent algorithm”, namely the row coloring primitives discussed in this post, but uses “interpolation/pattern matching” to approximately detect the cutoff points.
When I started this work, I hoped to find more clearly increasing or decreasing embedding circuits dictating the cutoff points, which would be interpretable without falling back to “pattern matching”. (This was the inspiration for adding X and Y embeddings in Section 5. Resulting curves are not as smooth as I’d hoped). I think the next step (not sure if I will do this) might be to continue training this network, either simply for longer, with smaller batches, or with the entire input set (not holding about half out for testing) to see if resulting curves become smoother.
--
This thought was inspired by a short email discussion I had with Marius Hobbhahn, one of the authors of the original solution. I have his permission to share content from our email exchange here. Marius wants me to “caveat that [he, Marius] didn’t spend a lot of time thinking about [my original post], so [any of his thoughts from our email thread] may well be wrong and not particularly helpful for people reading [this comment]”. I’m not sure this caveat just adds noise since this thought is mine (he has not commented on this thought) and I don’t currently think it is worthwhile to summarize the entire thread (and the caveat was requested when I initially asked if I could summarize our entire thread), so not sharing any of his thoughts here, but I want to respect his wishes even if this caveat mostly (or solely) adds noise.
Thank you for the kind words and the offer to donate (not necessary but very much appreciated). Please donate to https://strongminds.org/ which is listed on Charity Navigator’s list of high impact charities ( https://www.charitynavigator.org/discover-charities/best-charities/effective-altruism/ )
I will respond to the technical parts of this comment tomorrow or Tuesday.
Enjoyed this post! Quick question about obtaining the steering vectors:
Do you train them one at a time, possibly adding an additional orthogonality constraint between each train?