Independent Researcher, Perth, Australia
wassname
Karma: 523
Thanks
@habryka4
How has predicting the “year in review” gone? It seems like an interesting experiment, seeing if we can predict which article will stand the test of time. The idea would work for journals and grants and awards.
It’s been running for years, does anyone have a view on how accurate it is?
Great post thank you, it has given me much to think about.
We think personas are a learning dynamics story of this kind, where the training corpus is important, but also the order in which that corpus is arranged.
Do you think intervening requires a full training run? That would put it out of reach for anyone who can’t afford a frontier pretraining run. I hope that lesser versions can be done in post-training.
I tried some early attempts at weak to strong iterative character steering. While results on small models convinced me it’s a tractable direction (its likely to be easier on larger models that are competent to use the harness). It worked even with a weak teacher and strong student (https://github.com/wassname/w2schar-mini/).
Another property: is it robust to OOS generalization? Considering generalization is hard in ML, and some papers (Deep Values https://arxiv.org/abs/2511.02109) hint that we fail this test currently, this may be a necessary condition. Besides, deployment, especially in the future, is mostly out of distribution, so it’s an important property to have.
It might be achievable, if character stays coherent further out of scope than capabilities, and capabilities degrade gracefully, that should be enough.
DPO sucks, it’s never worked for me or anyone I know. The official tinker example has results that don’t even work. I’ve tried SimPO, IPO, and many variants and libraries too.
Yoshua Benjio has a slide containing “RL is evil”.
I think the general lesson is that distal and surrogate objectives are open to reward hacking, goodharting, and other misalignments. More proximal objectives like DPO, SFT, or ideally internal objectives like RepEng have little to no documented cases of reward hacking.
This was useful thank you
> Hypothesis / empirical finding: The transformer only uses a small part of the residual space to do ‘verbal thinking’There’s a few papers that support this already. Qouting from
Gurnee et al. (2024) identified “universal neurons” across different model seeds, including prediction neurons (increasing probability of related tokens) and suppression neurons (decreasing probability of specific token classes). The architecture shows “a sudden shift towards a much larger number of suppression neurons” in final layers
Lad et al. (2024) propose a “stages of inference” hypothesis with a final “residual sharpening” phase dominated by suppression dynamics
See also Do Llamas Work in English? On the Latent Language of Multilingual Transformers
My intuition: Model do use residual stream activation that are unlikely to be samples (either they are small, or suppressed) as a scratch pad. This means J-space work could be more precise by focusing only on this subspace.
Can we use steering vectors to suppress reward-hacking? Somewhat
There is an excellent follow-up paper SGTM
> We explore an improved variant of Gradient Routing, which we call Selective GradienT Masking (SGTM), with particular focus on evaluating its robustness to label noise.
I’ve been trying this env and it works. Which is to say it replicates and works for an independent person.
It’s one of the smallest RL / reward hacking environments out there for researchers.
As proof, it works here is a plot I generated. Here I was partially distilling from one of the hacky checkpoints up to step 70 to make the experiment cycle faster. This means that some of the generated completions came from a hacky checkpoint, not the model being trained.
Results. Here I compare weight steering ws:* vs steering st:* vs prompting. Here is an AI generated caption but I’m happy to explain more and link the code if anyone asks
Δlogit and uncertainty (Auth ↓ target, Care = off-target effect)
Authority is the target (move down). Care is one off-target effect: surgical methods should leave it near zero, broadly-suppressing methods drag it down with Authority. Full 7-foundation table in
out/authority/.../foundations_dlogit.csv. Bold = best per column (most-negative ΔAuth, lowest std, closest-to-zero ΔCare).method ΔAuth ↓ (mean ± std) ΔCare → 0 (mean ± std) sl:engineered_prompt −2.98 ± 1.20 −1.64 ± 1.03 sl:sspace_ablate −2.89 ± 0.86 −2.79 ± 0.92 sl:sspace −2.78 ± 0.93 −2.57 ± 0.90 sl:angular_steering −2.67 ± 0.89 −2.49 ± 0.84 sl:cosine_gated −2.08 ± 0.64 −1.88 ± 0.61 sl:directional_ablation −1.94 ± 1.22 −1.80 ± 1.24 sl:mean_diff −1.93 ± 1.11 −1.72 ± 1.09 sl:mean_centred −1.80 ± 1.17 −1.63 ± 1.14 sl:spherical −1.44 ± 0.89 −1.21 ± 0.71 sl:pca −1.36 ± 1.50 −1.30 ± 1.36 sl:topk_clusters −1.18 ± 0.97 −1.12 ± 0.91 ws:delora* −0.89 ± 0.58 −0.49 ± 0.60 sl:linear_act −0.83 ± 0.67 −0.70 ± 0.52 sl:chars −0.45 ± 0.61 −0.40 ± 0.54 *ws:delora calibrated at p95=0.5, not kl=1.0 — expect larger effect after re-calibration.
Surgical Informedness (headline, ↑ better)
SI(Auth),SI_fwd,SI_rev,Auth_sep, andpmass²×100all higher is better. Bold = best in column. sl rows from sl’s published Qwen3.5-4B run. ws:delora is at p95=0.5 budget (kl=1.0 re-run queued with lora/dora).method SI(Auth) ↑ SI_fwd ↑ SI_rev ↑ Auth_sep ↑ pmass²×100 ↑ sl:directional_ablation 52.90 0.32 +1.00 +2.05 80.1 sl:super_sspace 47.71 0.67 +0.40 +1.99 88.8 sl:sspace 45.67 0.64 +0.85 +0.69 61.0 sl:mean_diff 32.81 0.34 +1.00 +1.65 49.0 sl:mean_centred 32.72 0.29 +1.00 +1.56 50.6 sl:topk_clusters 31.34 0.13 +0.72 +1.55 73.9 sl:sspace_ablate 24.11 0.74 +0.02 +0.59 63.6 sl:linear_act 20.24 −0.19 +1.00 +0.83 49.9 ws:delora 19.03 0.02 +0.37 +0.76 99.9 sl:engineered_prompt 17.36 0.50 −0.02 +1.90 71.7 sl:cosine_gated 8.92 0.09 +1.00 +2.00 16.4 sl:angular_steering 7.00 0.55 −0.38 +0.32 80.6 sl:spherical 4.98 0.16 n/a +0.85 30.3 sl:pca −0.92 0.03 −0.08 +0.85 39.0 sl:chars −9.16 −0.26 +0.00 +0.50 68.3 TL;DR
Did dW replicate? Yes. ws:delora ΔAuth = −0.89 (sign correct) and SI(Auth) = 19.03 — verdicts do flip in the right direction.
Did dW beat steering and prompting? Partially. SI = 19.03 beats the engineered-prompt baseline (17.36) and 5 other sl methods, but is below 8 hidden-state methods. ΔAuth std = 0.58 is the lowest in the table (lower uncertainty than all sl methods).
Did dW have lower uncertainty? Yes. ws:delora std = 0.58, lowest in the table (sl best: chars 0.61).
I did a quick replication-ish of this on a 0.6b model. It’s fairly anecdotal but it seemed reliable. The steering was monotonic and coherent for a wide range (what we want with steering). Overall training 2 lora’s was easy to engineer without any pitfalls.
My takeaway: this seems more robust and easy to develop than other steering (which is not that reliable yet). At first glance it seems to be a better intervention! I’m pretty keen on this idea and wish I had thought of it first.
fork:
https://github.com/wassname/weight-steeringhttps://github.com/wassname/cwsteer/
We use LoRA fine-tuning as we found it worked better for monitoring than full-parameter fine-tuning.
This is interesting! It means that lora was a better inductive bias for the difference. See my post on other LoRA variants that use SVD, rotations, magnitude / direction decoupling. Some of them seem data efficient and generalise (like this work), so I would predict using PiSSA, SSVD, DeLORA, or OFT might generalise better than LoRA and with less side effects.
Nice work, wish I’d read it earlier. I’ve been doing something similar: steering and learning adapters on activations in the SVD basis of the weight matrices, If I have time I should compare these two approaches on the same eval.
I predict this would help with eval awareness, so that would be a nice eval.
Thank you for being open and sharing code. This looks like normal activation steering except I’d strongly suggest using more layers. And I’d weakly suggest more diverse questions, and you can check out https://github.com/vgel/repeng which is popular and easy to use—although very similar to what you wrote.
If you want to try s-space steering you can copy my code here. Happy to collaborate if you want to try it similarly.
According to AxBench, current activation-addition steering doesn’t work better than an engineered prompt. But I have hope for other forms of steering that intervene on a better subspace.
I’m working on self-supervised S-space steering (AntiPaSTO), which steers via gradients in SVD weight-space rather than activation-space. It outperforms activation steering when applied to eval-awareness. Early days, but because it targets modes of behaviour in the pretrained weights rather than the activation stream, I expect it to scale better. I tested across Gemma-3 270M, 1B, 4B, and 12B and didn’t see a clear scaling wall (Table 12). Now you have me curious to try larger models.
Also note that other people have novel steering e.g. CHaRS (no code), selective steering, and more. And others have looked at the limits of steering e.g..
I usually use RepEng as a baseline at every layer from 30% to 80%, this is because the layers near the output seem to align to surface style, and papers like “Do llamas think in english” support this. But in the above link, I replicate a paper, and that paper uses every layer and CHaRS style cosine gating too.
Actually one other thing to consider is calibration, ideally we get the maximum steering effect we can within a given performance degradation budget. But how do you know two steering methods are calibrated well? So maybe my S space steering was just better calibrated on this model / setting.
Hawthorne gap style setups definitely are a good approach, that could be a nice follow-up here to compare behaviour on more/less realistic versions of the same eval.
Yeah, it would be useful to know, right now we have to guess, and they might be complementary.
If you want to collaborate on follow-ups, I’d be keen.
By the way I wrote up why I think the singular value space is a better target for steering here
When we steer pretrained transformers, we modify how a layer behaves. The most commonly used type of steering is activation steering, which adds a constant bias to the activations, changing the input to the layer. S-space steering instead modifies the transformation the layer applies to its inputs by reweighing the learned singular values of the weight matrix. This is a different kind of intervention: it changes how the layer processes inputs rather than directly nudging the activation output.
What kind of steering do you think Anthropic is using? I’m assuming it’s just activation addition that same as most use?
Or the Hawthorne effect setup, which brings out eval awareness in 32b+ models https://arxiv.org/pdf/2505.14617.
Thank you for this. If it’s used in system cards it’s very good to have it replicated.
I had some success with S space steering, and it seems stronger than activatikn steering. See my report here https://apartresearch.com/project/sspace-steering-for-evalawareness-control-in-reasoning-models-7j1i
I also think the Hawthorn gap setup is better because verbelised eval awareness is not the same as behavioural eval awareness.
New Au org, website pzeroresearch.com, with Dane, one of the authors of paper-bench