Tinkerer with interpretability for alignment. Research @ Argonne, UChicago, Meta, and Georgia Tech.
Pranav Viswanath
Karma: 30
Thanks for the kind words Joseph, appreciate the read!
Congrats @wesg on the huge win for interpretability! In the paper, you asked:
“The J-lens… accounts for less than a tenth of the overall activity in Claude’s internal processing. So what is all the rest of the neural network doing?”
You indirectly answer this by ablating the J-space to measure the effect of the rest—I read the non-J space directly by using Anthropic’s NLAs on Llama-70B, and found that it’s causal even when the model flatly denies it.
I split concept vectors into their J-space/non-J parts, and ran Lindsey’s “Introspection Awareness” experiment on both halves.
The model denies any non-J injection, but the NLA reads it! Here’s some examples:
And the non-J side isn’t a weak residual—it drives the injected token’s logprob up +21 nats while the model names it 0% of the time and the NLA ~100%. It’s causal, but the model is blind to it.
I even stress-tested this with “Franken-vectors” gluing one concept’s subconscious half to another’s conscious half. The model only ever names the J-side concept; the NLA reads both!
This gets at @Neel Nanda’s skepticism from his review, flagging that he wasn’t “confident enough in NLAs to be confident the ground truth in the case studies is correct.”
The injection setup gives that ground truth by construction: known concept in, and the NLA names it with zero false positives across four controls (no-injection, random-in-J, random-out-of-J, mislabeled). But this is on synthetic injections, so the natural next step is applying on natural activation scenarios like line counting and eval awareness.
I’m also interested in your take on how fixed the J-rank should be—I found conscious reportability scales smoothly as we scale rank (as non-J space goes dark), so the mental workspace doesn’t have a sharp cutoff. Curious whether the boundary you drew is based on reportability, causality, or something else.
If interested, I go more in depth about this J-space x NLA study in my LW writeup with code on GitHub. Thank you and congrats again!
I used Anthropic’s NLA to catch the thoughts Llama70B can’t report outside its J-space! New post here.
I split concept vectors (“secrecy”, “elephant”, “democracy”...) into J-space and non-J parts, and injected them separately using Lindsey’s “Introspection Awareness” experiment.
The model immediately reports the J-space thought, and is totally blind to the non-J space.
But even when the model denies it, an NLA catches the hidden thoughts, still steering its behavior!
Happy to chat extensions to the experiment and alignment implications :)
Some other points I’m curious for anyone’s take on:
What decides reportability? The non-J residual clearly carries causal weight and the majority of the injected concept variance, so I wonder what gates have to pass to actually surface into the mental workspace.
Would an Activation Oracle (or other activation verbalizers) fare better here? I think the unsupervised high FVE reconstruction is what makes NLAs good by design at catching the rest of the activation, would be curious to think pros/cons for other interp tools.
Is there a more deterministic way to find the J-space boundary? I had to use rank sweeps that yielded the cleanest results, would be cool to come up with an expression as a function of the dim-size or some other known quantity.
Nice job all! Very cool that this layer-wise backprop makes the early layers more interpretable.
Saw you did the MLP gain study in the appendix, did you also look for the attention gain? (section 4.3.2 in the J space paper).
If R lens vectors have higher broadcast across tokens than J vectors, it’s evidence that these are better workspace candidates!