jacob_drori

• jacob_drori 11 Aug 2025 22:19 UTC

  2 points

  0

  in reply to: Leon Lang’s comment on: The Cod­ing The­o­rem — A Link be­tween Com­plex­ity and Probability

  Ah I hadn’t noticed that, very nice. Great post!

• jacob_drori 11 Aug 2025 17:53 UTC

  3 points

  0

  on: The Cod­ing The­o­rem — A Link be­tween Com­plex­ity and Probability

  In case anyone else didn’t know what it meant for a set of binary strings to be “prefix-free”, here’s Claude’s explanation, which I found helpful:
  

  A set of binary strings is prefix-free if no string in the set is a prefix of any other string in the set.

  Example:

  • ✅ Prefix-free: {0, 10, 110, 111}

  • ❌ Not prefix-free: {0, 01, 10} (because “0” is a prefix of “01″)

  Why does this matter for Turing machines?

  The key is in how universal Turing machines work. A universal machine U simulates any other machine T by receiving input of the form i′q, where:

  • i′ = prefix-free encoding of machine T’s description

  • q = the actual input to feed to machine T

  U must parse this concatenated input to figure out: “Where does the machine description end and the actual input begin?”

  Without prefix-free encoding: Given input “00101”, U can’t tell if the machine description is “0“, “00”, “001”, etc. - there’s ambiguity.

  With prefix-free encoding: Once U reads a complete machine description, it knows exactly where that description ends and the input begins. No ambiguity, no delimiters needed.

  This unique parseability is essential for universal machines to correctly simulate other machines, and it’s crucial for Kolmogorov complexity theory where we need to measure program lengths precisely without parsing ambiguity.
  

• jacob_drori 3 Aug 2025 20:44 UTC

  28 points

  5

  on: ja­cob_drori’s Shortform

  Operationalizing the definition of a shard

  Pope and Turner (2022) define a shard as follows:

  A shard of value refers to the contextually activated computations which are downstream of similar historical reinforcement events.

  To operationalize their definition, we must decide exactly what we mean by contextually activated computations, and by similar reinforcement events. One could philosophize here, but I’ll just pick somewhat-arbitrary definitions and run with them, for the sake of quickly producing something I could turn into code.

  Following Apollo/​Goodfire, I will identify contextually activated computations with certain directions in parameter space. These directions might be found using APD, L3D, SPD, or some future method.

  I am not sure when to consider two reinforcement events (i.e. RL reward-assignments) “similar”. But if we replace “reinforcement events” with “RL updates”, there is a natural definition: cluster RL parameter updates, and call two updates similar if they are in the same cluster. A given update should be allowed to be in multiple clusters, and we suspect parameter updates enjoy some linear structure. Therefore, a natural clustering method is an SAE.^[1] Then, a shard is a decoder vector of an SAE trained on RL parameter updates.

  Annoyingly, parameter vectors much larger than the activation vectors that SAEs are usually trained on, posing three potential issues:

  1. Storing enough parameter updates to train our SAE on takes a ton of memory.

  2. The SAE itself has a ton of parameters—too many to effectively train.

  3. It may be impossible to reconstruct parameter updates with reasonable sparsity (with $L_0\sim 100$, say, like standard SAEs).

  To mitigate Issue 1, one could focus on a small subset of model parameters. One could also train the SAE at the same time as RL training itself, storing only a small buffer of the most recent parameter updates.

  L3D cleverly deals with Issue 2 using low-rank parametrizations for the encoder and decoder of the SAE.

  Issue 3 may turn out not to occur in practice. Mukherjee et al (2025) find that, when post-training an LLM with RL, parameter updates are very sparse in the neuron basis. Intuitively, many expect that such post-training merely boosts/​suppresses a small number of pre-existing circuits. So perhaps we will find that, in practice, one can reconstruct parameter updates with reasonable sparsity.

  All of this is to say: maybe someone should try training an SAE/​applying L3D to RL parameter updates. And, if they feel like it, they could call the resulting feature vectors “shards”.

  1. ^

     For each SAE latent, the corresponding cluster is the set of data points on which that latent is active.

  2. ^

     Shard theory is supposed to apply to humans, too, but here I focus on NNs since humans seem more complicated.

ja­cob_drori’s Shortform

• jacob_drori 1 Aug 2025 17:47 UTC

  9 points

  0

  on: ja­cob_drori’s Shortform

  Some issues with the ICL Superposition paper
  
  Setup:

  Xiong et al (2024) show that LLMs can in-context-learn several tasks at once. Consider e.g. the following prompt:

  France -> F
  Portugal -> Portuguese
  Germany -> Berlin
  Spain -> Madrid
  Russia -> R
  Poland -> Polish
  Italy ->

  A model will complete this prompt sometimes with Rome, sometimes with I, and sometimes with Italian, learning a “superposition” of the country -> capital, country -> first-letter and country -> language tasks. (I wish they hadn’t used this word: the mech interp notion of superposition is unrelated).

  Let $D_i$ be the proportion of the in-context examples that correspond to task $i$, and let $P_i$ be the probability that the model completes the prompt according to task $i$. Call a model calibrated if $P_i\approx D_i$: the probability it assigns to a task is proportional to the number of times the task appeared in-context. A measure of the degree of calibration is given by the KL-divergence:

  $KL\left(D||P\right)={\sum }_i{D}_i\log \left(D_i/P_i\right).$

  Lower KL means better calibration.

  Issue 1:

  The authors show that, for a certain set of 6 tasks, and with 60 in-context examples, calibration improves with model size:
  

  

  The best reported KL between the uniform distribution [0.167, 0.167, …, 0.167] and the model’s output distribution is around 1.5. To give a sense for what this means, here are four examples of distributions with such a KL:

  $P_1=[0.004,0.004,0.004,0.33,0.33,0.33]$

  $P_2=[0.02,...,0.02,0.9]$

  $P_3=[{10}^{-5},0.19999,...,0.19999]$

  $P_4=[0.004,0.008,0.016,0.032,0.2,0.74]$.

  Seeing these distributions convinced me that the models they studied are not well-calibrated. I felt a bit misled by the paper.

  Issue 2:

  Say we take turns with the model, repeatedly appending a new “question” (e.g. a country) to the prompt and allowing the model to generate an “answer” (e.g. the country’s capital, language, or first letter) at default temperature 1.0. The authors state:

  After the first token is generated, the model tends to converge on predicting tokens for a single task, effectively negating its ability for multi-task execution.

  But this sort of mode-collapse behavior is not what we would observe from a well-calibrated model!

  For simplicity, consider the case of only two tasks: $A$ and $B$. Let $n_A$ and $n_B$ be the number of examples of each task, prior to a given generation step. A calibrated model generates $A$ with probability $\frac{n_A}{n_A+n_B}$, in which case $(n_A,n_B)\leftarrow (n_A+1,n_B)$. Otherwise, it generates $B$, and $(n_A,n_B)\leftarrow (n_A,n_B+1)$.

  This is precisely the Pólya urn model. If the numbers of each task are initially $A_0$ and $B_0$, and then we generate for a long time, it can be shown that the limiting proportion of $A$ tasks is a random variable $X$ distributed as follows:

  $$P[X=x]\propto x^{A_0-1}(1-x{)}^{B_0-1}.$$

  In the realistic setting where $A_0,B_0>1$, this distribution is peaked around$\frac{A_0-1}{A_0+B_0-2}$, i.e. roughly the proportion of the initial tasks that were $A$, and goes to zero at $x=0$ or $1$. So calibrated models don’t collapse; they tend to maintain roughly the initial ratio of tasks.

• jacob_drori 1 Aug 2025 14:15 UTC

  9 points

  1

  in reply to: Neel Nanda’s comment on: Op­ti­miz­ing The Fi­nal Out­put Can Obfus­cate CoT (Re­search Note)

  My intuition at least for the single word case is that saying a word increases the probability the model will say it in future. Therefore, if it wants the probability it says that the future to be low, it’s incentivised not to say it anywhere.

  
  I share your intuition, and think you are describing what we called “conditional spillover”.^[1] The fact that the Mind+Face intervention did not fully prevent spillover in our experiments is some evidence that conditional spillover is occurring.

  One might mitigate conditional spillover as follows. In REINFORCE, the usual parameter update is (setting batch size = lr = 1 for brevity, and being a little schematic):

  $\delta \theta =r\left(\underset{g_{\text{CoT}}}{\underset{}{{\nabla }_{\theta }\log p_{\theta }(\text{CoT}}}\right)+\underset{g_{\text{out}}}{\underset{}{{\nabla }_{\theta }\log p_{\theta }\left(\text{output | CoT}\right)}})$.

  Here, $r$ is the reward, which depends only on the output. In the case where $r$ is the sum of a task-completion and an obfuscation-inducing term (as in our current work), we can write:

  $\delta \theta =(r_{\text{task}}+r_{\text{obfusc}})(g_{\text{CoT}}+g_{\text{out}})$.

  In order to mitigate conditional spillover, we can simply modify the gradient update by removing the $r_{\text{obfusc}}\times g_{\text{CoT}}$ term:

  $(\delta \theta {)}_{\text{modified}}=r_{\text{task}}{g}_{\text{CoT}}+(r_{\text{task}}+r_{\text{obfusc}})g_{\text{out}}$.

  If this mitigation is used in combination with Mind+Face, then the Mind is not reinforced for producing CoTs which increase the probability of output obfuscation. It is only reinforced for task completion.

  1. ^

     Side note: I don’t love our “conditional spillover” and “parametric spillover” terminology, since they sound like different observed effects, whereas they are really two different (proposed) mechanisms underlying the same observed effect. If brevity weren’t an issue, I’d call them “spillover due to later tokens conditioning on earlier ones” and “spillover due to parameter sharing between the models generating CoT and output”, or something.

• jacob_drori 31 Jul 2025 20:29 UTC

  4 points

  0

  in reply to: TheManxLoiner’s comment on: Book Re­view: The MANIAC

  IIRC, When We Cease… was part fictional, but presented itself as fact. I remember being pretty annoyed when I eventually realized this (though possibly it was obvious all along and I was just being dumb).

• jacob_drori 31 Jul 2025 9:24 UTC

  5 points

  0

  in reply to: Daniel Kokotajlo’s comment on: Op­ti­miz­ing The Fi­nal Out­put Can Obfus­cate CoT (Re­search Note)

  Mind/​Face requires more compute than generating with a single model, since 1) one must keep both sets of model weights in memory, and 2) when the face “takes over” generation, it must recompute the entire KV-cache up to that token.
  
  However, Mind/​Face takes no more than 2x the compute compared to standard generation (since you can always use a naive approach: just run forward passes of both models in parallel at each generation step, and choose which model’s predicted token to append conditional on whether we are in thinking or output mode). I expect this upper bound to be loose. We haven’t thought much about optimizations yet, but as Luke mentioned, using a smaller model for the Face (whose job is easier) seems reasonable.

• jacob_drori 30 Jul 2025 23:21 UTC

  8 points

  0

  in reply to: ryan_greenblatt’s comment on: Op­ti­miz­ing The Fi­nal Out­put Can Obfus­cate CoT (Re­search Note)

  Agreed. Thanks for the useful clarification.

  In future work, we may try to produce a demo where optimizing only the output leads to steganographic CoT. But I expect this to be fairly challenging on a practical level (evidence: Roger and McCarthy et al had difficulty inducing steganography via RL. But note that I’m not up to date on this literature). We’d appreciate suggestions for RL environments where producing such a demo may be feasible.

Op­ti­miz­ing The Fi­nal Out­put Can Obfus­cate CoT (Re­search Note)

SAE on ac­ti­va­tion differences

• jacob_drori 29 Jun 2025 3:17 UTC

  1 point

  0

  on: Sparsely-con­nected Cross-layer Transcoders

  It may be possible to massively reduce memory usage in sparsely-connected mode.

  Let $B$ be batch size, $K$ be num active latents per dictionary per token, and $F$ be num latents per dictionary.

  My current implementation of sparsely-connected mode has a terrible $O\left(F^2\right)$ memory usage, since each virtual weight matrix has $F^2$ elements. But how many of these virtual weights do we actually need to compute?

  Upstream latents: On each token in the batch, we only need the virtual weights connecting to the $K$ active upstream latents.

  Downstream latents: Strictly speaking, we should compute activations for every downstream latent, since we don’t know in advance which will be active. But, insofar as vanilla mode closely approximates sparsely-connected mode, we should be okay to only compute virtual weights connecting to downstream latents that were active in vanilla mode.

  So on each token, we only need to compute $K^2$ virtual weights, and so the memory requirement is $B{K}^2$, which is small.

  Of course, this new approach loses something: sparsely-connected mode now relies on vanilla mode to tell it which latents should activate. So much for a standalone replacement model! I think a reasonable middle-ground is to only compute virtual weights to the $100\times K$ (say) latents with largest vanilla preactivation. Then compute sparsely-connected preactivations for all those latents, and apply TopK to get the activations. The memory usage is then $100B{K}^2$ which is still small.

  What links here?

  • Sparsely-con­nected Cross-layer Transcoders by jacob_drori (18 Jun 2025 17:13 UTC; 45 points)

• jacob_drori 23 Jun 2025 8:27 UTC

  2 points

  0

  on: The Crois­sant Prin­ci­ple: A The­ory of AI Generalization

  Related: https://​​arxiv.org/​​abs/​​2004.10802

• jacob_drori 19 Jun 2025 15:47 UTC

  1 point

  0

  in reply to: williawa’s comment on: Sparsely-con­nected Cross-layer Transcoders

  The approach you suggest feels similar in spirit to Farnik et al, and I think it’s a reasonable thing to try. However, I opted for my approach since it produces an exactly sparse forward pass, rather than just suppressing the contribution of weak connections. So no arbitrary threshold must be chosen when buikding a feature circuit/​attribution graph. Either two latents are connected, or they are not.
  
  I also like the fact that my approach gives us sparse global virtual weights, which allows us to study global circuits—something Anthropic had problems with due to interference weights in their approach.

Sparsely-con­nected Cross-layer Transcoders

• jacob_drori 27 Mar 2025 17:14 UTC

  1 point

  0

  on: Me­moriza­tion-gen­er­al­iza­tion in practice

  Vary temperature t and measure the resulting learning coefficient function $\lambda \left(t\right)$

  This confuses me. IIUC, $p_{\beta ,n}\left(w\right)=\frac{\phi \left(w\right)\exp (-n\beta L_n(w\left)\right)}{\int d{w}^{\prime }\phi \left(w^{\prime }\right)\exp (-n\beta L_n(w^{\prime }\left)\right)}$. So changing temperature is equivalent to rescaling the loss by a constant. But such a rescaling doesn’t affect the LLC.
  
  What did I misunderstand?

• jacob_drori 7 Mar 2025 16:26 UTC

  2 points

  1

  in reply to: Lucius Bushnaq’s comment on: Es­ti­mat­ing the Prob­a­bil­ity of Sam­pling a Trained Neu­ral Net­work at Random

  Let $V\left(ϵ\right)$ be volume of a behavioral region at cutoff $ϵ$. Your behavioral LLC at finite noise scale is $\lambda \left(ϵ\right):=d\log V/d\log ϵ$, which is invariant under rescaling $V$ by a constant. This information about the overall scale of $V$ seems important. What’s the reason for throwing it out in SLT?

• jacob_drori 27 Feb 2025 6:03 UTC

  3 points

  2

  in reply to: Daniel Tan’s comment on: Emer­gent Misal­ign­ment: Nar­row fine­tun­ing can pro­duce broadly mis­al­igned LLMs

  Fantastic research! Any chance you’ll open-source weights of the insecure qwen model? This would be useful for interp folks.

• jacob_drori 26 Feb 2025 22:03 UTC

  8 points

  0

  on: [PAPER] Ja­co­bian Sparse Au­toen­coders: Spar­sify Com­pu­ta­tions, Not Just Activations

  The Jacobians are much more sparse in pre-trained LLMs than in re-initialized transformers.

  This would be very cool if true, but I think further experiments are needed to support it.

  Imagine a dumb scenario where during training, all that happens to the MLP is that it “gets smaller”, so that MLP_trained(x) = c * MLP_init(x) for some small c. Then all the elements of the Jacobian also get smaller by a factor of c, and your current analysis—checking the number of elements above a threshold—would conclude that the Jacobian had gotten sparser. This feels wrong: merely rescaling a function shouldn’t affect the sparsity of the computation it implements.

  To avoid this issue, you could report a scale-invariant quantity like the kurtosis of the Jacobian’s elements divided by their variance-squared, or ratio of L1 and L2 norms, or plenty of other options. But these quantities still aren’t perfect, since they aren’t invariant under linear transformations of the model’s activations:

  E.g. suppose an mlp_out feature F depends linearly on some mlp_in feature G, which is roughly orthogonal to F. If we stretch all model activations along the F direction, and retrain our SAEs, then the new mlp_out SAE will contain (in an ideal world) a feature F’ which is the same as F but with larger activations by some factor. On the other hand, the mlp_in SAE should will contain a feature G’ which is roughly the same as G. Hence the (F, G) element of the Jacobian has been made bigger, simply by applying a linear transformation to the model’s activations. Generally this will affect our sparsity measure, which feels wrong: merely applying a linear map to all model activations shouldn’t change the sparsity of the computation being done on those activations. In other words, our sparsity measure shouldn’t depend on a choice of basis for the residual stream.

  I’ll try to think of a principled measure of the sparsity of the Jacobian. In the meantime, I think it would still be interesting to see a scale-invariant quantity reported, as suggested above.

• jacob_drori 23 Jan 2025 17:56 UTC

  1 point

  0

  on: Against blan­ket ar­gu­ments against interpretability

  We have pretty robust measurements of complexity of algorithms from SLT

  This seems overstated. What’s the best evidence so far that the LLC positively correlates with the complexity of the algorithm implemented by a model? In fact, do we even have any models whose circuitry we understand well enough to assign them a “complexity”?

  … and it seems like similar methods can lead to pretty good ways of separating parallel circuits (Apollo also has some interesting work here that I think constitutes real progress)

  Citation?