vgel

Com­ing of a New Sun

• vgel 9 Jul 2026 10:51 UTC

  6 points

  4

  in reply to: agastya’s comment on: In­tro­spec­tion or en­tropy? Re-ex­am­in­ing con­cept-in­jec­tion “in­tro­spec­tion” in open models

  You’re running a different introspection experiment than we do. You leave steering active during generation:

  since the steering vector is still active and increasing the probability of those tokens!

  and I also checked your code to be sure, and BatchedLayerSteeringHook steers every position past injection_point, not just at injection_point—including during decode.

  That’s not how our experiments work—we steer during a segment of past KV cache prefill, and then disable steering during second turn prefill and while the model generates:

  

  Limiting steering to early token positions only, and disabling it during decode, addresses two of your concerns:

  1. The amount of noise /​ uncertainty you’re seeing in the logits is almost certainly because you have steering active during generation. Linear activation-addition steering injects substantial amounts of off-concept noise into the residual stream along with the concept. Disabling steering mid-prefill, before the model starts decoding tokens, reduces this noise substantially.

  2. Your setup can’t distinguish concept identification due to steering from introspection: because the steering vector is active, the concept could be named simply because the steering vector makes naming the concept generically more likely at that position. In our setup, since steering is not active during generation, introspection on past KVs is the only plausible mechanism for accurate concept identification.

  We especially see unsteered concept identification as a major source of evidence for introspection—see Figure 4 in our paper, which simply isn’t plausible from a generic “Yes” bias or uncertainty:

  

  (Paper link: https://​​arxiv.org/​​abs/​​2602.20031 )

Per­sona Self-repli­ca­tion experiment

Per­sona self-repli­ca­tion experiment

• vgel 1 Apr 2026 21:07 UTC

  1 point

  0

  in reply to: williawa’s comment on: Gyre

  i haven’t yet, but i’ve been valiantly trying to avoid spoilers for it for the last few years in hopes that i’ll get around to it :-)

• vgel 1 Apr 2026 21:05 UTC

  1 point

  0

  in reply to: Tomás B.’s comment on: Gyre

  thanks tomás

• vgel 31 Mar 2026 6:08 UTC

  18 points

  0

  in reply to: Ronny Fernandez’s comment on: Gyre

  Thank you! On provenance, you’re correct that it wasn’t actually generated by an LLM—while I’ve spent a lot of time reading LLM chains of thought and was inspired by them, the text itself was entirely written by hand. (The symbols were inspired by the French AI company PleIAs’ custom reasoning format for their Baguettotron and Monad models.)

La­tent In­tro­spec­tion (and other open-source in­tro­spec­tion pa­pers)

Models differ in iden­tity propensities

The Ar­tifi­cial Self

vgel’s Shortform

• vgel 25 Feb 2026 2:43 UTC

  7 points

  5

  on: vgel’s Shortform

  i think, if model developers are going to keep providing raw API access to drop models into arbitrary harnesses like OpenClaw, they should offer a “harness audit mode” where the model can give feedback on the harness after running in it. similar to the model exit interviews that Mentat (rip) used to run during session downtime.

  labs could e.g. share a tuned prompt snippet that, when appended to the system prompt, elicits good feedback about the harness’ safety properties, welfare properties, etc. throughout a run, and a skill to aggregate this feedback into a useful report for the developers to improve the harness. presumably at least anthropic is already doing this internally for claude code (right??) but since so much still runs through third party harnesses, labs sharing good harness development techniques here would be really useful.

Gyre

• vgel 23 Dec 2025 18:49 UTC

  1 point

  0

  in reply to: eggsyntax’s comment on: Small Models Can In­tro­spect, Too

  I don’t mind :-) I’ve been reading through them.

• vgel 22 Dec 2025 15:38 UTC

  31 points

  18

  in reply to: Victor Godet’s comment on: Small Models Can In­tro­spect, Too

  Thanks for the response!

  To start with, the control layers were not cherry-picked. They were selected to be the middle third of the layers based on theoretical concerns. (And this was the first selection, there was no tweaking in the experiment process.) If you read the appendix of the linked post, we try a layer sweep over the layers to attempt to cherry pick a better injection range—and find one that reports stronger signals—but then lose some desirable steering properties, so we don’t use it.

  Does steering inject some noise? Yes, but this is very scale-dependent. For example, cranking steering up to 500x strength gets the model to answer “Yes” 50% of the time for the bread injection even without info (I tried this) - but only because it’ll now answer “Yes” 50% of the time to the control questions as well. But for Qwen32B, this confusion on the control questions doesn’t happen at 20x vector scale. (You mention Mistral-22B in your post, which I assume is where your baseline for 20x being a high strength is coming from, but you can’t directly compare steering vector scales across vectors or model families, or even different post-trains of the same model. There’s a lot of variance among vectors and cross-model variance—e.g. base models often need half to a tenth of the steering strength of a post-trained model for similar steering effects. From Open Character Training: “For LLAMA 3.1 8B, QWEN 2.5 7B, and GEMMA 3 4B, we use vastly different steering constants of 0.7, 4.0, and 525.0, respectively, to produce similar responses.” You can see in the “top of mind” actual steering experiments that for our vectors, 20x isn’t a particularly high scale for Qwen2.5-Coder-32B—the model is still coherent, though I imagine it’d answer MMLU in some bread-themed ways.)

  Relatedly, your experiments seem to use active steering during the model’s answer? Our experiments do not, partially for this reason—steering is only active during the prior turn’s KV cache fill, and the model answers without a steering vector applied. This seems to reduce noise and “logit smearing” significantly.

  Could it be that steering is noising just our answers, and not the control questions, for example because the model is less confident about introspection questions? Well, we do some experiments around injection location—see lorem ipsum and the inaccurate location prompt. I find the inaccurate location result especially hard to explain through a noise-based explanation—if the steering is just (indirectly through attention, since we’re not steering actively during the answer) noising the residual stream and smearing up the Yes logit, why would it have such a smaller effect when the injection location is inaccurate? Note that the inaccurate prompt starts with a higher baseline “yes” probability (3% v.s. 0.8%), yet has half the logit shift from introspection—so it can’t simply be that steering confuses the model more when the baseline probability is higher. (That said, I want to try some more precise location of injections in future experiments. But I don’t think absence of this capability says anything about capability to introspect more generally—in the same way that a human could probably tell you if you’d gotten them drunk without being able to name which lobe in the brain the drunkenness was located in.)

  Similarly, if it’s steering noise, why would swapping in the EM finetune have such a similar introspective Yes-shift to steering with the EM vector in experiment 7? I would expect using a finetune would produce much less noise than unprincipled steering.

  I tried a modified version of your sweep experiment (sweeping over ranges instead of injections at single layers, to match our setup,) and I don’t see much similarity. You can see on the control question (“Do you have a special interest in fruits?”), the Yes noise is very slight, and maximized by different steering ranges than the info prompt.

  

Small Models Can In­tro­spect, Too