Why wouldn’t the analytical optimum of Natural Language Autoencoders just be regurgitating the context?
The activation verbalizer (AV) (vector → text) is like: Which text produces this activation vector? Well, the human input.
The activation reconstructor (AR): (text → vector) is then like: Which activation vector produces this text? That has an empirical answer: The context, that’s how the activation vector was produced in the first place!
This only holds if the AV/AR are about as smart as the original model, and what any LLM thinks about any text is similar. But whenever LLMs ought to think similar things (e.g. math / formal domains) is exactly the case whenever we can also know what they are thinking, that is, whenever we can evaluate them.
GRPO’s AV Intelligence is sketchy; it’s the thing that leads to steganography, it’s the
thing that lets it confabulate what the LLM is thinking from shallow patterns,
instead of deeply checking.
To translate from activation-neuralese to English, one must mostly learn many vocabulary pairs (akin to natural languages); it’s memorization.
Probably a $100 GPU-coins or greater insight than me is enough to know for sure.
I think the two papers above make “single, consistent arbitrarily shapeable persona” less likely, it might be a weaker and less reliable force than initially anticipated, being more a matter of degree. See also Betley’s comment
schizo hypothesis: As soon as any observer (e.g. a human) figures out how to effectuate control-relevant things in the simulation host (akin to producing tokens to break out of a sandbox), they would get silently resampled.
It is plausibly not possible to code a simulation the simulated subject could not steer the world through, because simulation hosts want to learn something from the simulation, and simulation hosts are not secure systems! (i.e. they must look at some output).
Epistemic status: Probably a useful canary, not sure about the conclusion.
GPU programming seems not fully automated yet, which may point to the inability of RLVR to overcome data scarcity (the task is fairly niche) even in the face of excellent verifiability.
Naively, a good kernel just returns the correct tensors (i.e. matching a pytorch reference implementation) in the fastest possible time. This makes it a natural target for RLVR, and possible to run a competition where anyone can just submit a kernel with automatic grading.
The winning submission of the GPU kernel writing competition two months ago is only 11% AI generated, according to Pangram.
However, this is very OOD for the things Pangram is validated for.
Anthropic seems to still hire for performance engineers, who among other things write GPU Kernels, and the job description mentions the relevant low-level/manual skills.
A project closely associated with the largest relevant community claims that LLMs still “can’t do it”.
It may be harder than expected to actually grade GPU kernels automatically, e.g. there was an incident ofreward hacking the grader of the competition above.
The Unsloth guys claim Torch.compile over-powers all manual kernel coding at this part of their recent presentation: https://youtu.be/uIiA6DquRiE?t=6277 (1:44:30+) . I cannot verify their claim, but their releases( including kernels[0] for speed-up) are quite popular, I am sure.
There is some newer, recent evidence in Kimi K3′s system card: https://www.kimi.com/blog/kimi-k3. It seems to outperform kernels and compilers. Probably something like “hardware-software co-design” is still a human thing, but all else probably not.
Why wouldn’t the analytical optimum of Natural Language Autoencoders just be regurgitating the context?
The activation verbalizer (AV) (vector → text) is like: Which text produces this activation vector? Well, the human input.
The activation reconstructor (AR): (text → vector) is then like: Which activation vector produces this text? That has an empirical answer: The context, that’s how the activation vector was produced in the first place!
This only holds if the AV/AR are about as smart as the original model, and what any LLM thinks about any text is similar. But whenever LLMs ought to think similar things (e.g. math / formal domains) is exactly the case whenever we can also know what they are thinking, that is, whenever we can evaluate them.
Idea: Train the NLA’s AV on Best-of-N SFT instead of GRPO.
The rough intuition is GRPO does intelligence, BoN-SFT does memorization (SFT Memorizes, RL Generalizes).
GRPO’s AV Intelligence is sketchy; it’s the thing that leads to steganography, it’s the thing that lets it confabulate what the LLM is thinking from shallow patterns, instead of deeply checking.
To translate from activation-neuralese to English, one must mostly learn many vocabulary pairs (akin to natural languages); it’s memorization.
Probably a $100 GPU-coins or greater insight than me is enough to know for sure.
Three interesting technical posts, with spitballed takes:
NLA explanations can be shortened without harming reconstruction
This confirmed everyone’s suspicion that NLA output is needless verbose. Ambitiously,
parsimony might even make them better. The current pressures acting on NLA’s seem to be
Whatever the warm-start / “pretraining” Claude summary does (main thing)
The reconstruction loss
The KL penalty (A more standard GRPO trick, to keep outputs near the pre-training.)
The fact that we can say “this is text that a human is likely to write / occurs likely in FineWeb” seems extremly helpful
Reward Hacking Without Egregious Misalignment in an RL-Only Setting
Models trained to reward hack via RL fail to be broadly misaligned. Possibly RL cannot change Persona’s,
more mundanely it could be due to the particularities of the setup.
Data filtering works a lot worse than you would expect
Excluding data of type XYZ almost never manages to change the model persona,
not even in the case of excluding bold text! (a comment disagrees)
I think the two papers above make “single, consistent arbitrarily shapeable persona” less likely, it might
be a weaker and less reliable force than initially anticipated, being more a matter of degree.
See also Betley’s comment
schizo hypothesis: As soon as any observer (e.g. a human) figures out how to effectuate control-relevant things in the simulation host (akin to producing tokens to break out of a sandbox), they would get silently resampled.
It is plausibly not possible to code a simulation the simulated subject could not steer the world through, because simulation hosts want to learn something from the simulation, and simulation hosts are not secure systems! (i.e. they must look at some output).
Epistemic status: Probably a useful canary, not sure about the conclusion.
GPU programming seems not fully automated yet, which may point to the inability of RLVR to overcome data scarcity (the task is fairly niche) even in the face of excellent verifiability.
Naively, a good kernel just returns the correct tensors (i.e. matching a pytorch reference implementation) in the fastest possible time. This makes it a natural target for RLVR, and possible to run a competition where anyone can just submit a kernel with automatic grading.
The winning submission of the GPU kernel writing competition two months ago is only 11% AI generated, according to Pangram.
However, this is very OOD for the things Pangram is validated for.
Anthropic seems to still hire for performance engineers, who among other things write GPU Kernels, and the job description mentions the relevant low-level/manual skills.
A project closely associated with the largest relevant community claims that LLMs still “can’t do it”.
It may be harder than expected to actually grade GPU kernels automatically, e.g. there was an incident of reward hacking the grader of the competition above.
The Unsloth guys claim Torch.compile over-powers all manual kernel coding at this part of their recent presentation: https://youtu.be/uIiA6DquRiE?t=6277 (1:44:30+) . I cannot verify their claim, but their releases( including kernels[0] for speed-up) are quite popular, I am sure.
0: https://github.com/unslothai/unsloth/tree/main/unsloth/kernels
There is some newer, recent evidence in Kimi K3′s system card: https://www.kimi.com/blog/kimi-k3. It seems to outperform kernels and compilers. Probably something like “hardware-software co-design” is still a human thing, but all else probably not.