> Hypothesis / empirical finding: The transformer only uses a small part of the residual space to do ‘verbal thinking’
There’s a few papers that support this already. Qouting from
Gurnee et al. (2024) identified “universal neurons” across different model seeds, including prediction neurons (increasing probability of related tokens) and suppression neurons (decreasing probability of specific token classes). The architecture shows “a sudden shift towards a much larger number of suppression neurons” in final layers
Lad et al. (2024) propose a “stages of inference” hypothesis with a final “residual sharpening” phase dominated by suppression dynamics
My intuition: Model do use residual stream activation that are unlikely to be samples (either they are small, or suppressed) as a scratch pad. This means J-space work could be more precise by focusing only on this subspace.
This was useful thank you
> Hypothesis / empirical finding: The transformer only uses a small part of the residual space to do ‘verbal thinking’
There’s a few papers that support this already. Qouting from
Gurnee et al. (2024) identified “universal neurons” across different model seeds, including prediction neurons (increasing probability of related tokens) and suppression neurons (decreasing probability of specific token classes). The architecture shows “a sudden shift towards a much larger number of suppression neurons” in final layers
Lad et al. (2024) propose a “stages of inference” hypothesis with a final “residual sharpening” phase dominated by suppression dynamics
See also Do Llamas Work in English? On the Latent Language of Multilingual Transformers
My intuition: Model do use residual stream activation that are unlikely to be samples (either they are small, or suppressed) as a scratch pad. This means J-space work could be more precise by focusing only on this subspace.