Francisco Ferreira da Silva

• Francisco Ferreira da Silva 29 Jul 2026 16:37 UTC

  4 points

  0

  in reply to: Davidmanheim’s comment on: Can Claude teach me to make coffee?

  I find this surprising! I’ve tried this with Opus 4.8 and Fable 5 and they both failed pretty spectacularly. While their vision appears significantly better than Sonnet’s was here, the failures of reasoning remain. To give an example that I found funny/​exasperating: it suggested I report that the espresso machine was broken to its owner (that’d be me), failing to realize that the reason it would not turn on is that what Claude told me to press (multiple times!) was not the power button.

Fran­cisco Fer­reira da Silva’s Shortform

• Francisco Ferreira da Silva 27 Jul 2026 10:07 UTC

  18 points

  0

  on: Fran­cisco Fer­reira da Silva’s Shortform

  Stefan Heimersheim and I recently introduced a measurement method that appears to distinguish feature directions from non-feature directions in LLMs, see here for details. tl;dr, We perturb activations into candidate feature directions and fit an -norm to the response function; corresponds to feature directions, to non-feature directions.

  
  We applied our method to J-lens directions from Anthropic’s recent workspace paper in Qwen3.6-27B (for which many of the key results have been replicated, see here for details) and found . This makes us think that J-lens directions are not aligned with the model’s features, and not privileged by its error correction mechanism. Given that our method appears to reliably separate feature directions from non-feature directions, this makes us update downwards on the trustworthiness of J-lens. On the other hand, given that J-lens seems otherwise good, this also makes us update downwards on the reliability of our method.

  
  Thanks to Neel Nanda for suggesting this experiment.

  
  Experimental details:

  1. We perturb at layer 36. Qwen3.6-27B has 64 layers, workspace layers are a band ~30–90% through the model, and our method works best on directions far from the model’s output.

  2. Our analysis runs on pairs of directions, with each token corresponds to a direction as per J-lens. We need low cosine similarity within a pair, because we make them orthogonal for our analysis. If they’re not already close to orthogonal we distort them significantly, which changes results in unpredictable ways. We employ a threshold of .

  3. We run our analysis for 100 Chinese pairs of directions corresponding to 200n distinct tokens, drawn from a pool of 1200 single-token Chinese words selecting for lower cosine similarity. All pairs come out at .

  4. We tried doing the same or English, but it was a struggle to find J-lens directions with low cosine similarity for English tokens. We relaxed the threshold to 0.25 to reach because at 0.1 only one pair survived. We are not sure why this was hard for English but not for Chinese tokens.

  5. We compute the median and a bootstrapped 99% confidence interval for it, shown in the plot below together with contrastive feature directions. These do have , consistent with them being aligned with the model’s features.

  

  Code for reproduction: https://​​github.com/​​FranciscoHS/​​fsec-paper/​​tree/​​main/​​jlens
  
  

• Francisco Ferreira da Silva 16 Jul 2026 13:49 UTC

  2 points

  0

  in reply to: Alex Gibson’s comment on: Com­pressed Com­pu­ta­tion un­der L⁴ Loss is likely Com­pu­ta­tion in Superposition

  I have run it for higher values, yes. gives a very similar solution to . I just tried it for and it also appears to give something similar but with wider spread in codeword size. However, this appears to just be an optimization failure: the -trained network loses to the -trained network on the objective. I suppose the shape of the gradient for very high is just not great: it becomes dominated by the most extreme outliers giving the optimizer too noisy a signal. I haven’t thought about this too much, though.

• Francisco Ferreira da Silva 16 Jul 2026 9:31 UTC

  4 points

  0

  in reply to: Alex Gibson’s comment on: Com­pressed Com­pu­ta­tion un­der L⁴ Loss is likely Com­pu­ta­tion in Superposition

  Thank you!

  Let me restate your proposed setting first to make sure I follow.

  We want to compute , i.e., a scalar function that naively requires n layers of width d each to compute exactly.

  The network we are now interested in is one of width and depth -- so it can in principle trivially compute instances of in parallel.

  We are interested in a regime where we compute more than such instances, so we assume we compute (sparse) features simultaneously.

  Your proposal is then to divide the network into blocks of width , and assign a subset to feature $i$. These blocks correspond to the binary codewords in the post.

  I definitely agree that this works exactly if there are no collisions. This sounded too-good-to-be-true to me at first, but then I realized that’s because that’ll lead to quite a bit of leakage to inactive features—the construction in the post is also in principle capable of exactly computing an active feature in the collision-less case, but it turns out to be more favorable loss-wise to undershoot active features to reduce leakage to inactive features. I suspect this would also be the case if we managed to elicit a construction like this through training.

  This was a long-winded way of saying: I agree.

Com­pressed Com­pu­ta­tion un­der L⁴ Loss is likely Com­pu­ta­tion in Superposition

Ev­i­dence for fea­ture-spe­cific er­ror cor­rec­tion in LLMs

• Francisco Ferreira da Silva 23 Mar 2026 13:57 UTC

  2 points

  0

  in reply to: Tim Hua’s comment on: Find­ing fea­tures in Trans­form­ers: Con­trastive di­rec­tions elicit stronger low-level per­tur­ba­tion re­sponses than baselines

  Thank you! Yes, we have. I actually generated a plot comparing steering effectiveness to perturbation response a few weeks back, but it seems LessWrong does not allow images in comments, so I’ll just describe it :) Steering effectiveness is somewhat input and concept dependent, but for the English-to-Mandarin case in Qwen3, we observe a peak of ~80% of output Mandarin tokens for a perturbation angle of ~20 degrees. This aligns quite well with the peak of the perturbation response curve; similarly, there is good alignment between the perturbation magnitude at which we start to observe changes in output and at which the perturbation response becomes non-zero.

Find­ing fea­tures in Trans­form­ers: Con­trastive di­rec­tions elicit stronger low-level per­tur­ba­tion re­sponses than baselines

• Francisco Ferreira da Silva 15 Mar 2026 9:45 UTC

  3 points

  2

  on: The Ar­tifi­cial Self

  The arXiv link at the top currently links to the gradual disempowerment preprint.

• Francisco Ferreira da Silva 11 Mar 2026 11:19 UTC

  7 points

  5

  in reply to: Gordon Seidoh Worley’s comment on: Don’t Let LLMs Write For You

  I agree with this directionally, but would add a caveat to your research paper caveat. If it’s a research paper of the type ‘I ran experiment X and found Y’ then yes, I suppose the words don’t matter so much beyond conveying the result—although I’ve found Claude to have a tendency to overclaim and overinterpret results that needs to be carefully kept in check, to the extent that I often think “man it’d have been easier to write this myself!”

  But ‘I ran experiment X and found Y’ is not what every research paper is! For many research papers, I definitely care about the ideas of the author and how they convey them, so I’d rather they stay mostly LLM-free.