TL;DR: Multi-head Latent Attention (MLA), the attention variant behind DeepSeek-V2/V3, compresses all of a layer’s keys and values through a single shared bottleneck before splitting them back out per head. I trained a small (114M-param) MLA transformer from scratch, pretrained on a general web/code/math mixture, then fine-tuned on TinyStories and poked at what that bottleneck actually does, analyzing it in this fine-tuned state. Four things stood out: it learns to store content and throw away position almost perfectly, on their own, without being told to; all the induction heads in the model set up shop at a single layer instead of spreading out; one layer looks like a “hub” by two independent measures at once; and the bottleneck is, on average, only half full. All of this is from one small model, so treat it as “here’s an interesting pattern, someone should check if it holds up” rather than “here’s how MLA works.” I’ll flag the uncertainty as I go rather than saving it for a disclaimer at the end.
Full writeup with all the tables and methodological details is arxiv:2607.23054, 2026. This post is the shorter, more opinionated version.
Why bother with MLA specifically
If you’ve been keeping half an eye on efficient transformer architectures, you’ve probably run into the fact that KV-cache size is one of the main things that makes long-context LLM inference expensive—you have to store a key and value vector for every past token, for every head, for every layer, and that adds up fast. MLA’s fix, introduced with DeepSeek-V2, is to project each token’s keys and values from a much smaller shared vector (they call it cKV, 128-dim in my model, vs. 512-dim residual stream) instead of computing them independently per head. Only that small compressed vector needs to be cached. DeepSeek reports an 81% reduction in KV-cache size from this trick, and it’s now sitting inside a 671B-parameter production model.
What nobody seems to have asked, as far as I could find, is: what does that compressed vector actually contain? It’s a genuine bottleneck − 128 dimensions is a lot less room than 512 - so something has to get thrown away. The architecture suggests an answer (position info is supposed to route around the bottleneck via RoPE, not through it), but “the architecture suggests” and “the trained model actually does this” are different claims, and I wanted to check the second one.
Setup, briefly
24-layer, 114M-param MLA transformer. Pretrained for 16,500 steps on a mix of general web text, code, and math (60% FineWeb-Edu, 25% The Stack, 15% OpenWebMath), then fine-tuned for 7,200 steps on TinyStories (simple children’s stories, small vocab, simple syntax). Every analysis below runs on the model in this fine-tuned state, using TinyStories-style text as the analysis input, so the representations were shaped by a fairly broad pretraining mixture, but everything I measured reflects how the model behaves once specialized to the simpler TinyStories domain. cKV is 128-dim, sitting inside a 512-dim residual stream. All the plots below come from one training run of this one model. I’ll come back to why that matters at the end, but I want to flag it now too: nothing here has been checked across seeds, and I don’t have a matched standard-attention (MHA) model trained the same way to compare against. Where I say “unlike standard attention,” I’m leaning on Olsson et al.’s induction-head paper, which studied different, larger models, not a controlled comparison.
With that said, here’s what I found.
Finding 1: the bottleneck learns to be content-only, on its own
If you train linear probes on cKV to predict different properties of each token, entity identity (is this token part of a character’s name?) comes out at 95.5% accuracy, statistically indistinguishable from probing the full 512-dim residual stream (also 95.5%). The compressed 128-dim vector loses nothing about entity identity, at least nothing a linear probe can find.
Position is the opposite story. A probe trying to recover which of 16 position-bins a token is in gets 21.7% from cKV, barely above the 6.25% chance rate, versus 37.2% from the residual stream (which itself isn’t great, but is clearly better than cKV).
This held at every layer I checked, not just one; entity accuracy stays close to the ceiling from layer 0 to layer 23, and position accuracy stays near chance the whole way through.
The architecture has a separate pathway for positional information (RoPE, injected directly, bypassing cKV), so in principle the model could have used cKV for position too; nothing stops gradient descent from putting redundant information there if it were useful. It didn’t. That’s a real, if modest, finding: given the option to be lazy and dump everything into the shared bottleneck, the model instead learned a clean division of labor that matches what the architecture’s authors probably intended but never (as far as I know) verified empirically.
(Small honesty note: I initially had a stale results table with worse numbers than this. An old run’s data hadn’t been swapped out for the current one. Just mentioning it because it’s exactly the kind of thing worth double-checking before you trust anyone’s, including your own, reported numbers.)
Finding 2: all the induction heads live on one layer
Induction heads are the mechanism a transformer uses to do “if I’ve seen this pattern before, copy what came next.” For example, if the text earlier said “Alice was...”, and we’re now at ”...Alice was”, an induction head pushes probability toward whatever followed the first occurrence. They’re a big deal in interpretability because they’re one of the few circuits that’s been traced end-to-end and tied to in-context learning.
I scored every one of the 192 attention heads in my model (24 layers × 8 heads) for induction behavior. Every single head with a meaningfully high induction score sits at layer 12. Not “clustered around” layer 12, but exactly at layer 12, specifically heads 2, 3, 5, 6, and 7.
In the original induction-heads work (Olsson et al., on standard multi-head attention), induction heads showed up scattered across multiple layers. My best guess for why MLA is different: keys and values for every head at a given layer are derived from the same compressed vector. If you’re an induction head, you need consistent key/value info to do your copying, and it might just be architecturally cheaper for the circuit to converge on one layer’s shared bottleneck rather than try to stay coordinated across several separately-compressed layers. That’s a hypothesis, not something I’ve directly tested. I don’t have, say, an ablation that shows the co-location breaks if you decouple the KV compression per head.
Finding 3: one layer looks like a hub, by two different measures
Two experiments I ran completely independently both pointed at layer 15:
An SVD of the KV-compression matrix at each layer shows layer 15 using the most of its 128-dim capacity (88 effective dimensions, versus a 59-dim average across all layers).
A corruption experiment (adding noise to early tokens, seeing how much that changes cKV at each layer, and weighting that by how much it hurt the model’s prediction) shows layer 15 with the strongest signal of any layer.
Two unrelated measurements landing on the same layer is the kind of thing that makes you sit up. But I want to be precise about what the second measurement actually shows, because I initially described it more strongly than it deserves (this is worth saying honestly rather than glossing over). What I ran is not the classic “activation patching” technique, where you surgically restore clean activations into a corrupted run and measure how much output gets fixed. I measured something cheaper. In the writeup I call it disruption attribution: how much a layer’s cKV changes under corruption, scaled by how much the prediction got worse. That’s a correlational signal (“this layer’s representation shifts when things go wrong”) rather than a direct causal test (“restoring this layer’s representation fixes things”). It’s a reasonable proxy, and I think the layer-15 finding is a real pattern worth someone following up on, but “semantic hub” is currently an inference from correlational data, not something I’ve causally nailed down. If I get compute for a follow-up, implementing genuine patch-and-restore is the first thing on the list.
Finding 4: the bottleneck is only about half full
Given all 128 dimensions are available, how many does the model actually use? I took the SVD of the compression matrix at each layer and asked how many singular values you need to capture 99% of the variance. Average across layers: 59 out of 12, 46%. At a 90% threshold it’s even starker, ~17 dimensions.
This isn’t uniform across layers, either. There’s a rough anti-correlation where layers using more of their KV budget use less of their Q budget, and vice versa, as if the model is doing some kind of dynamic allocation between “what to attend to” and “what’s attending.”
If this held up at production scale, it suggests you could probably get away with a smaller, non-uniform bottleneck, bigger for the layers that need it (like 15), smaller everywhere else, and save more KV-cache memory on top of what MLA already saves. I want to be upfront that I haven’t tested this; it’s a natural next experiment (fine-tune a version with heterogeneous rank and check the perplexity hit), not something I’ve done.
What I’d want someone to check before believing this generalizes
Putting all the caveats in one place, even though I tried to flag them inline above too:
One model, one training run. I don’t know which of these findings would survive a different random seed. The specific layer numbers (12, 15) especially, I’d bet those move around across seeds even if the qualitative pattern (heads co-locate somewhere, some layer becomes a hub) holds.
No matched baseline. The “unlike standard attention” claim about induction heads leans on a different paper studying different models. A same-size, same-data MHA model run through the same analysis is the obvious missing control.
Narrow analysis domain, broader pretraining: The model was pretrained on a fairly diverse mix (web text, code, math), so it’s not purely a “trained on children’s stories” toy model. But every experiment above runs on the fine-tuned checkpoint using TinyStories-style inputs, and I haven’t checked whether these patterns hold if I probe with FineWeb-Edu-, code-, or math-style text instead. TinyStories’ simplicity may still be inflating how clean the entity-tracking numbers look.
~2,000-5,900x smaller than the models this is meant to be informative about: DeepSeek-V2 (which introduced MLA) is 236B params, about 2,000x mine. DeepSeek-V3 is 671B, about 5,900x mine. Circuit structure is known to shift with scale in other contexts, so I’d hold these conclusions loosely when thinking about what happens inside actual DeepSeek models.
None of this means I think the findings are wrong; I think the patterns are real in this model, and I think the four experimental angles (SVD, head taxonomy, probing, corruption-based attribution) are a reasonable toolkit for anyone who wants to check whether they hold up elsewhere. I just don’t want to oversell a 114M-parameter model, fine-tuned on TinyStories, as a definitive account of an architecture running in production at up to ~5,900x the scale.
If anyone wants to replicate this at a larger scale, or with a matched MHA baseline, or push on the heterogeneous-rank idea, I’d be very happy to compare notes. That’s a large part of why I’m posting this rather than sitting on it until I have more compute.
What does DeepSeek’s attention bottleneck actually keep? A small mechanistic study of MLA
TL;DR: Multi-head Latent Attention (MLA), the attention variant behind DeepSeek-V2/V3, compresses all of a layer’s keys and values through a single shared bottleneck before splitting them back out per head. I trained a small (114M-param) MLA transformer from scratch, pretrained on a general web/code/math mixture, then fine-tuned on TinyStories and poked at what that bottleneck actually does, analyzing it in this fine-tuned state. Four things stood out: it learns to store content and throw away position almost perfectly, on their own, without being told to; all the induction heads in the model set up shop at a single layer instead of spreading out; one layer looks like a “hub” by two independent measures at once; and the bottleneck is, on average, only half full. All of this is from one small model, so treat it as “here’s an interesting pattern, someone should check if it holds up” rather than “here’s how MLA works.” I’ll flag the uncertainty as I go rather than saving it for a disclaimer at the end.
Full writeup with all the tables and methodological details is arxiv:2607.23054, 2026. This post is the shorter, more opinionated version.
Why bother with MLA specifically
If you’ve been keeping half an eye on efficient transformer architectures, you’ve probably run into the fact that KV-cache size is one of the main things that makes long-context LLM inference expensive—you have to store a key and value vector for every past token, for every head, for every layer, and that adds up fast. MLA’s fix, introduced with DeepSeek-V2, is to project each token’s keys and values from a much smaller shared vector (they call it cKV, 128-dim in my model, vs. 512-dim residual stream) instead of computing them independently per head. Only that small compressed vector needs to be cached. DeepSeek reports an 81% reduction in KV-cache size from this trick, and it’s now sitting inside a 671B-parameter production model.
What nobody seems to have asked, as far as I could find, is: what does that compressed vector actually contain? It’s a genuine bottleneck − 128 dimensions is a lot less room than 512 - so something has to get thrown away. The architecture suggests an answer (position info is supposed to route around the bottleneck via RoPE, not through it), but “the architecture suggests” and “the trained model actually does this” are different claims, and I wanted to check the second one.
Setup, briefly
24-layer, 114M-param MLA transformer. Pretrained for 16,500 steps on a mix of general web text, code, and math (60% FineWeb-Edu, 25% The Stack, 15% OpenWebMath), then fine-tuned for 7,200 steps on TinyStories (simple children’s stories, small vocab, simple syntax). Every analysis below runs on the model in this fine-tuned state, using TinyStories-style text as the analysis input, so the representations were shaped by a fairly broad pretraining mixture, but everything I measured reflects how the model behaves once specialized to the simpler TinyStories domain. cKV is 128-dim, sitting inside a 512-dim residual stream. All the plots below come from one training run of this one model. I’ll come back to why that matters at the end, but I want to flag it now too: nothing here has been checked across seeds, and I don’t have a matched standard-attention (MHA) model trained the same way to compare against. Where I say “unlike standard attention,” I’m leaning on Olsson et al.’s induction-head paper, which studied different, larger models, not a controlled comparison.
With that said, here’s what I found.
Finding 1: the bottleneck learns to be content-only, on its own
If you train linear probes on cKV to predict different properties of each token, entity identity (is this token part of a character’s name?) comes out at 95.5% accuracy, statistically indistinguishable from probing the full 512-dim residual stream (also 95.5%). The compressed 128-dim vector loses nothing about entity identity, at least nothing a linear probe can find.
Position is the opposite story. A probe trying to recover which of 16 position-bins a token is in gets 21.7% from cKV, barely above the 6.25% chance rate, versus 37.2% from the residual stream (which itself isn’t great, but is clearly better than cKV).
This held at every layer I checked, not just one; entity accuracy stays close to the ceiling from layer 0 to layer 23, and position accuracy stays near chance the whole way through.
The architecture has a separate pathway for positional information (RoPE, injected directly, bypassing cKV), so in principle the model could have used cKV for position too; nothing stops gradient descent from putting redundant information there if it were useful. It didn’t. That’s a real, if modest, finding: given the option to be lazy and dump everything into the shared bottleneck, the model instead learned a clean division of labor that matches what the architecture’s authors probably intended but never (as far as I know) verified empirically.
(Small honesty note: I initially had a stale results table with worse numbers than this. An old run’s data hadn’t been swapped out for the current one. Just mentioning it because it’s exactly the kind of thing worth double-checking before you trust anyone’s, including your own, reported numbers.)
Finding 2: all the induction heads live on one layer
Induction heads are the mechanism a transformer uses to do “if I’ve seen this pattern before, copy what came next.” For example, if the text earlier said “Alice was...”, and we’re now at ”...Alice was”, an induction head pushes probability toward whatever followed the first occurrence. They’re a big deal in interpretability because they’re one of the few circuits that’s been traced end-to-end and tied to in-context learning.
I scored every one of the 192 attention heads in my model (24 layers × 8 heads) for induction behavior. Every single head with a meaningfully high induction score sits at layer 12. Not “clustered around” layer 12, but exactly at layer 12, specifically heads 2, 3, 5, 6, and 7.
In the original induction-heads work (Olsson et al., on standard multi-head attention), induction heads showed up scattered across multiple layers. My best guess for why MLA is different: keys and values for every head at a given layer are derived from the same compressed vector. If you’re an induction head, you need consistent key/value info to do your copying, and it might just be architecturally cheaper for the circuit to converge on one layer’s shared bottleneck rather than try to stay coordinated across several separately-compressed layers. That’s a hypothesis, not something I’ve directly tested. I don’t have, say, an ablation that shows the co-location breaks if you decouple the KV compression per head.
Finding 3: one layer looks like a hub, by two different measures
Two experiments I ran completely independently both pointed at layer 15:
An SVD of the KV-compression matrix at each layer shows layer 15 using the most of its 128-dim capacity (88 effective dimensions, versus a 59-dim average across all layers).
A corruption experiment (adding noise to early tokens, seeing how much that changes cKV at each layer, and weighting that by how much it hurt the model’s prediction) shows layer 15 with the strongest signal of any layer.
Two unrelated measurements landing on the same layer is the kind of thing that makes you sit up. But I want to be precise about what the second measurement actually shows, because I initially described it more strongly than it deserves (this is worth saying honestly rather than glossing over). What I ran is not the classic “activation patching” technique, where you surgically restore clean activations into a corrupted run and measure how much output gets fixed. I measured something cheaper. In the writeup I call it disruption attribution: how much a layer’s cKV changes under corruption, scaled by how much the prediction got worse. That’s a correlational signal (“this layer’s representation shifts when things go wrong”) rather than a direct causal test (“restoring this layer’s representation fixes things”). It’s a reasonable proxy, and I think the layer-15 finding is a real pattern worth someone following up on, but “semantic hub” is currently an inference from correlational data, not something I’ve causally nailed down. If I get compute for a follow-up, implementing genuine patch-and-restore is the first thing on the list.
Finding 4: the bottleneck is only about half full
Given all 128 dimensions are available, how many does the model actually use? I took the SVD of the compression matrix at each layer and asked how many singular values you need to capture 99% of the variance. Average across layers: 59 out of 12, 46%. At a 90% threshold it’s even starker, ~17 dimensions.
This isn’t uniform across layers, either. There’s a rough anti-correlation where layers using more of their KV budget use less of their Q budget, and vice versa, as if the model is doing some kind of dynamic allocation between “what to attend to” and “what’s attending.”
If this held up at production scale, it suggests you could probably get away with a smaller, non-uniform bottleneck, bigger for the layers that need it (like 15), smaller everywhere else, and save more KV-cache memory on top of what MLA already saves. I want to be upfront that I haven’t tested this; it’s a natural next experiment (fine-tune a version with heterogeneous rank and check the perplexity hit), not something I’ve done.
What I’d want someone to check before believing this generalizes
Putting all the caveats in one place, even though I tried to flag them inline above too:
One model, one training run. I don’t know which of these findings would survive a different random seed. The specific layer numbers (12, 15) especially, I’d bet those move around across seeds even if the qualitative pattern (heads co-locate somewhere, some layer becomes a hub) holds.
No matched baseline. The “unlike standard attention” claim about induction heads leans on a different paper studying different models. A same-size, same-data MHA model run through the same analysis is the obvious missing control.
Narrow analysis domain, broader pretraining: The model was pretrained on a fairly diverse mix (web text, code, math), so it’s not purely a “trained on children’s stories” toy model. But every experiment above runs on the fine-tuned checkpoint using TinyStories-style inputs, and I haven’t checked whether these patterns hold if I probe with FineWeb-Edu-, code-, or math-style text instead. TinyStories’ simplicity may still be inflating how clean the entity-tracking numbers look.
~2,000-5,900x smaller than the models this is meant to be informative about: DeepSeek-V2 (which introduced MLA) is 236B params, about 2,000x mine. DeepSeek-V3 is 671B, about 5,900x mine. Circuit structure is known to shift with scale in other contexts, so I’d hold these conclusions loosely when thinking about what happens inside actual DeepSeek models.
None of this means I think the findings are wrong; I think the patterns are real in this model, and I think the four experimental angles (SVD, head taxonomy, probing, corruption-based attribution) are a reasonable toolkit for anyone who wants to check whether they hold up elsewhere. I just don’t want to oversell a 114M-parameter model, fine-tuned on TinyStories, as a definitive account of an architecture running in production at up to ~5,900x the scale.
If anyone wants to replicate this at a larger scale, or with a matched MHA baseline, or push on the heterogeneous-rank idea, I’d be very happy to compare notes. That’s a large part of why I’m posting this rather than sitting on it until I have more compute.