KV-Cache Tensor Decomposition: Where Low-Rank Compression Is Mathematically Dead (and Where It Isn't)

Every few months a product announcement promises KV-cache compression "for free" — halve your cache, keep your quality. The mechanism is usually some flavor of low-rank approximation: project the cached keys and values into a smaller latent space, or share bases across heads or layers, and reconstruct on read. Sometimes the vendor publishes perplexity deltas that look fine; sometimes users report the compressed model quietly losing the thread three turns into a long conversation. A September 2026 math paper is the sledgehammer that settles which side of that divide you are on, because it stops asking "which compression algorithm is best" and asks the question that comes first: which axes of the KV cache are compressible at all?¹

The paper is Tensor Decomposition of Transformer Key-Value Caches: Spectral Structure and Format Comparison (arXiv:2609.28029, math.NA, submitted September 23, 2026, by Rahul Krishnan and Volker Schulz)¹. What it does is systematic and unusually honest: extract real KV caches from two production models — Mistral-7B-v0.3 (grouped-query attention, 32 layers, 8 KV heads, head dim 128) and LLaMA-2-13B (multi-head attention, 40 layers) — unfold the cache along each of its four axes, measure the singular-value spectrum of each unfolding, and then fit four standard tensor decompositions (Tucker, CP, tensor train, t-SVD) at matched storage. Every claim below is labeled: paper-verified where I read it in the paper, our-own-recompute where we verified the arithmetic, simulation where the number comes from a synthetic model built to isolate the mechanism.

The KV cache is a four-way tensor, and its axes are not equal

During autoregressive decoding, the KV cache at each layer holds, for every past token and every attention head, a key vector and a value vector. Stack the per-layer caches as slices and you get a fourth-order tensor whose axes are heads, tokens, features (the per-head feature dimension, 128 for both models studied), and layers — the paper groups adjacent layers, so the layer mode is the number of grouped layers, not all 32 or 40 at once². A tensor decomposition exploits correlations across several axes at once; which decomposition to use depends entirely on where those correlations actually live.

The paper's first result is a clean partition of the four axes into two classes — a "mode-count dichotomy"²:

• Token mode: compressible. The token-axis unfolding of the key cache drops by several orders of magnitude; substantial rank reduction is available at low error.
• Feature mode: compressible for keys, nearly incompressible for values. On keys the feature spectrum leaves room for truncation; on values it stays nearly flat.
• Head mode: effectively full-rank. The head unfolding requires its full rank at every error threshold the paper tests. Every head carries comparable energy; cutting heads costs real error — consistent with the per-head outlier structure that survives aggressive attention compression: each head is its own information channel, not a redundant copy of its neighbors.
• Layer (group) mode: effectively full-rank. Same picture across grouped layers — the smallest singular direction alone carries more energy than any practical error budget.

The paper calls the last two index-like (Definition 3.1 in the paper): a mode is index-like at error level ε if the smallest singular value of its unfolding carries more than ε² of the squared energy — in which case no approximation with that mode's rank truncated below full can achieve relative error ε. This is not a statement about any particular algorithm; it follows from the Eckart–Young–Mirsky theorem applied to the mode unfolding, and it is format-agnostic: the paper shows the measured forced-rank ablation errors sit within a few percent of this lower bound². No cleverness in the compressor escapes it.

Why this is an anti-hype sledgehammer

Now map that onto the product landscape. Cross-layer KV sharing, head-mixing projections, "merge eight heads into a latent" schemes — the entire family of "compress across heads or layers" claims is attempting rank reduction along axes whose measured spectra are flat. The paper's ablation makes the cost concrete: forcing the grouped-layer rank even one step below full (4 → 3 on Mistral groups) nearly triples the key reconstruction error, and halving the head rank produces a comparable jump. Worse, the forced configs barely improve as the storage budget relaxes from 2x to 4x compression — the truncated rank, not the budget, sets the error floor, and extra storage cannot buy it back³.

So when a vendor reports quality loss from a head- or layer-mixing compressor, that loss is not a bug in their implementation, not a matter of better calibration data or more training on the compression path. It is the flat spectrum of the head and layer modes asserting itself. The honest reading of the paper for buyers of compression systems:

1. Compress along tokens and features, keys first. That is where the low-rank structure lives.
2. Do not merge across heads or grouped layers at moderate error tolerances; the math says the loss is structural.
3. Keys and values need independent rank budgets — the value spectrum is flatter everywhere, and pooling them wastes key compressibility (a "K/V asymmetry" the paper traces back to the roles of the W_V and W_K projection matrices: values must spread energy across features to preserve content under arbitrary attention weights, keys need only a few discriminative directions for scoring⁴).
4. Compress keys pre-RoPE, not post-RoPE. The rotary embedding's position-dependent rotation flattens the token and feature spectra; post-RoPE keys lose 41%–64% of their pre-RoPE compressibility on both models⁴.

The caveat cuts both ways and the paper states it: all reconstruction sweeps use a single sequence length (1,024 tokens, WikiText-2 validation chunks), the spectra cover two models, and "effectively full-rank" is a statement at practical error levels, not an eternal constant. Extreme tolerances, different architectures, much longer contexts may differ; whether token-mode structure persists during incremental generation is explicitly left open³. Full-rank at 1% error is not full-rank at 50% error. But every "free 2x" claim on the market lives in exactly the error regime where the head and layer modes are certified incompressible.

Tucker wins because it can leave the full-rank modes alone

Given the spectra, the format comparison becomes almost overdetermined. The paper fits Tucker, CP, tensor train, and t-SVD to every layer of both models at matched storage across compression ratios from 2x to 5x. Result: Tucker achieves the lowest reconstruction error at every ratio on both models (CP was omitted on LLaMA-2-13B because matched-storage fitting exceeded the experiment GPU's memory; the LLaMA ordering is Tucker, then t-SVD, then TT)⁵.

The reason is structural, not a tuning accident. Tucker's multilinear ranks are per-mode and independently chosen — so the allocator can pin head and layer ranks at their full values, store nothing for those factors (a full-rank Tucker factor is the identity and can be omitted from storage), and spend the entire budget on the token and feature modes where the decay actually lives. CP forces every mode to share one rank, so compressing the tensor means truncating the head axis too. Tensor train threads a chain through all modes and pays a bond rank at every link. t-SVD applies truncated SVDs to frontal slices at every frequency, dragging head-mode structure into every truncation. All three are forced to spend budget — and pay error — on axes that have nothing to give.

And the paper doesn't leave the "just pin the full-rank modes" advice as folklore: Theorem 5.1, the mode-pinning theorem, gives a sufficient condition, checkable from the measured mode spectra alone (a worst-case exchange-cost argument against the mode's tail energy), under which the optimal Lagrangian rank allocator provably keeps a mode at full rank. On every tested head-mode configuration of both models across ratios 2x–5x, the certificate holds — the paper reports it satisfied on all 160 Mistral and all 224 LLaMA head-mode configurations, plus the large majority of grouped-layer cells⁶. This is the part practitioners should internalize: you can measure your own model's mode spectra (one SVD per unfolding, cheap) and certify in advance which axes a rank allocator will refuse to truncate, before fitting anything.

Keys and values want different formats

The paper's format sweep holds another asymmetry that only appears when you compare against two-dimensional baselines. Against per-head SVD and against cross-layer 2D factorizations in the style of Palu and xKV, the preferred representation splits on the K/V line⁷:

• Keys: 2D unfoldings win. The key tensor decays strongly along both big modes (tokens and features), so a matrix factorization that concentrates its whole budget on those two axes is most efficient. Cross-layer 2D (xKV-style) is lowest from 3x onward.
• Values: four-way Tucker wins. The value feature spectrum is nearly flat, so 2D factorizations hit their error floor; Tucker avoids it by sharing compact bases across heads and assigning independent per-mode ranks. Tucker-4D beats Tucker-3D and both 2D baselines at every tested ratio on values.

The practical takeaway is not "Tucker everywhere" but that the optimal cache format is axis- and tensor-dependent: 2D for keys, 4-way Tucker for values, full-rank pinned heads and layers in every case. A single compressor architecture applied identically to K and V is leaving quality on the table in one of the two.

Modeling the math: mixed-rank spectra and the cost of compressing a full-rank mode

The paper's spectra tables are measured on real caches from two checkpoints. Mistral-7B-v0.3 is open on HuggingFace, so its numbers are in principle reader-recomputable (the paper also points to its code repository)⁸; LLaMA-2-13B (meta-llama/Llama-2-13b-hf) sits behind Meta's license agreement — gated, so "recomputable from public weights" is true only for readers with approved access, and we say so plainly rather than pretend it is open.

To make the mechanism visible without downloading 15 GB of weights, here is a synthetic KV-like tensor built to have the paper's dichotomy: low-rank structure along tokens and features, full-rank (flat-spectrum) head and layer mixing. Everything below is a simulation — it isolates the linear algebra, it does not reproduce the paper's measured numbers.

import numpy as np
rng = np.random.default_rng(42)
 
# ---------- (a) synthetic KV-cache-like 4-way tensor ----------
H, S, F, L = 8, 512, 128, 4      # heads, tokens, features, grouped layers (Mistral-7B-v0.3-like)
r_token, r_feat = 16, 24        # low-rank token/feature structure; heads/layers left full-rank
T = np.empty((H, S, F, L))
for h in range(H):
    for l in range(L):
        Ut = rng.standard_normal((S, r_token)); Wt = rng.standard_normal((F, r_feat))
        C  = rng.standard_normal((r_token, r_feat)) / (1.0 + np.arange(r_token)[:, None])**2
        T[h, :, :, l] = (Ut @ C @ Wt.T) * (1.0 + 0.3 * rng.standard_normal())
T += 0.01 * rng.standard_normal(T.shape)
 
def rel_err(X, T): return np.linalg.norm(X - T) / np.linalg.norm(T)
 
def unfold(T, ax):
    return np.moveaxis(T, ax, 0).reshape(T.shape[ax], -1)
 
# ---------- (b) per-mode SVD spectra: the two classes ----------
print(f"tensor shape (heads, tokens, features, layer groups) = {T.shape}")
for name, ax in [('heads', 0), ('tokens', 1), ('features', 2), ('groups', 3)]:
    s = np.linalg.svd(unfold(T, ax), compute_uv=False)
    e = s**2 / (s**2).sum()
    cum = np.cumsum(e)
    r95, r99 = int(np.searchsorted(cum, 0.95) + 1), int(np.searchsorted(cum, 0.99) + 1)
    print(f"{name:9s} dim={T.shape[ax]:4d}  SV/SV1[:4]={np.round(s[:4]/s[0], 3)}  "
          f"rank(95%)={r95:3d}  rank(99%)={r99:3d}  min-SV energy={e[-1]:.5f}")

tensor shape (heads, tokens, features, layer groups) = (8, 512, 128, 4)
heads     dim=   8  SV/SV1[:4]=[1.    0.946 0.942 0.861]  rank(95%)=  8  rank(99%)=  8  min-SV energy=0.07563
tokens    dim=512  SV/SV1[:4]=[1.    0.938 0.831 0.804]  rank(95%)= 36  rank(99%)= 69  min-SV energy=0.00000
features  dim=128  SV/SV1[:4]=[1.    0.933 0.866 0.818]  rank(95%)= 32  rank(99%)= 57  min-SV energy=0.00001
groups    dim=   4  SV/SV1[:4]=[1.    0.854 0.83  0.825]  rank(95%)=  4  rank(99%)=  4  min-SV energy=0.21968

Read the last column. The head and group unfoldings need their full dimension to reach even 95% of the energy — the smallest singular values carry 7.6% and 22% of the energy respectively, so any truncation below full rank pays immediately. The token and feature unfoldings get 95% of the energy at rank 36 of 512 and 32 of 128. Two spectral classes, exactly as in the paper's Table 1 — here by construction, there by measurement.

Now the part that answers "why is compressing a full-rank mode not free": run a matched-storage sweep of ST-HOSVD (sequentially truncated Tucker) with two arms. In the pinned arm, head and layer factors stay full-rank — stored as omitted identity factors — and the budget goes entirely to the token and feature ranks. In the forced arm, the head rank is halved first (the "merge heads" move), and the allocator optimally redistributes the saved budget. Same storage budget in both arms.

# ---------- (c) matched-storage Tucker: head mode pinned vs halved ----------
def tucker_pin_tf(T, r_s, r_f):
    """ST-HOSVD truncating only token (mode 1) and feature (mode 2); heads+layers kept full."""
    M1 = np.transpose(T, [1, 0, 2, 3]).reshape(S, -1)
    U1 = np.linalg.svd(M1, full_matrices=False)[0][:, :r_s]
    core = (U1.T @ M1).reshape(r_s, H, F, L)
    M2 = np.transpose(core, [2, 0, 1, 3]).reshape(F, -1)
    U2 = np.linalg.svd(M2, full_matrices=False)[0][:, :r_f]
    c2 = (U2.T @ M2).reshape(r_f, r_s, H, L)
    X = np.transpose((U2 @ c2.reshape(r_f, -1)).reshape(F, r_s, H, L), [1, 2, 0, 3])
    X = (U1 @ X.reshape(r_s, -1)).reshape(S, H, F, L)
    return np.transpose(X, [1, 0, 2, 3])
 
def tucker_force_head(T, r_h, r_s, r_f):
    """Same budget, but head mode truncated to r_h first (spectral floor applies)."""
    M0 = T.reshape(H, -1)
    U0 = np.linalg.svd(M0, full_matrices=False)[0][:, :r_h]
    Tp = (U0.T @ M0).reshape(r_h, S, F, L)
    M1 = np.transpose(Tp, [1, 0, 2, 3]).reshape(S, -1)
    U1 = np.linalg.svd(M1, full_matrices=False)[0][:, :r_s]
    core = (U1.T @ M1).reshape(r_s, r_h, F, L)
    M2 = np.transpose(core, [2, 0, 1, 3]).reshape(F, -1)
    U2 = np.linalg.svd(M2, full_matrices=False)[0][:, :r_f]
    c2 = (U2.T @ M2).reshape(r_f, r_s, r_h, L)
    X = np.transpose((U2 @ c2.reshape(r_f, -1)).reshape(F, r_s, r_h, L), [1, 2, 0, 3])
    X = (U1 @ X.reshape(r_s, -1)).reshape(S, r_h, F, L)
    X = np.transpose(X, [1, 0, 2, 3])
    return np.einsum('ij,jsfl->isfl', U0, X)
 
orig = H * S * F * L
r_h = H // 2
print(f"\nmatched-storage sweep, original = {orig:,} scalars, head dim {H} forced to {r_h}")
for ratio in (2, 4, 8):
    budget = orig / ratio
    best_p = best_f = None
    for r_s in range(32, S + 1, 16):
        rf = (budget - r_s * S) / (r_s * H * L + F)      # pinned: core + 2 factors
        r_f = max(1, min(F, int(rf)))
        st = r_s * r_f * H * L + r_s * S + r_f * F
        if st > budget: continue
        e = rel_err(tucker_pin_tf(T, r_s, r_f), T)
        if best_p is None or e < best_p[0]: best_p = (e, (r_s, r_f), st)
    for r_s in range(32, S + 1, 16):
        for r_f in range(16, F + 1, 16):
            st = r_h * r_s * r_f * L + r_h * H + r_s * S + r_f * F  # + head factor
            if st > budget: continue
            e = rel_err(tucker_force_head(T, r_h, r_s, r_f), T)
            if best_f is None or e < best_f[0]: best_f = (e, (r_s, r_f), st)
    print(f"ratio {ratio:>2}x budget {int(budget):>7,}:  pinned (H,L full) err {best_p[0]:.4f}  "
          f"ranks {best_p[1]}  |  forced head={r_h} err {best_f[0]:.4f}  ranks {best_f[1]}")

matched-storage sweep, original = 2,097,152 scalars, head dim 8 forced to 4
ratio  2x budget 1,048,576:  pinned (H,L full) err 0.0130  ranks (224, 128)  |  forced head=4 err 0.6222  ranks (400, 128)
ratio  4x budget 524,288:  pinned (H,L full) err 0.0411  ranks (128, 108)  |  forced head=4 err 0.6222  ranks (192, 128)
ratio  8x budget 262,144:  pinned (H,L full) err 0.0837  ranks (80, 82)  |  forced head=4 err 0.6224  ranks (112, 96)

Two things to notice, both mirroring the paper's Section 5.3 ablation on real caches. First, pinning works: with head and layer modes left untouched, error creeps up gently as the budget tightens (1.3% → 8.4% from 2x to 8x), because the budget only ever truncates axes with decay to spare. Second, the forced arm is flat: 62.2% error at every budget, because once the head rank is below full, the dropped directions' energy is unrecoverable no matter how much token and feature rank you buy with the freed budget. The error floor is set by the rank, not the storage — the storage knob stops working. That flat line is the linear-algebra signature behind every disappointing head-mixing compressor demo.

One honesty note on the simulation: the forced arm benefits slightly from head-mixing being an orthogonal-ish projection here, which spreads the lost energy over surviving directions. Real caches whose heads carry independent content (the paper's index-like definition) hit the same floor through the same Eckart–Young tail energy. The numbers above are construction-dependent; the shape of the argument — flat floor vs gently rising curve — is not.

I considered a live probe of one real KV unfolding from Mistral-7B-v0.3 (range-reading the K projection via safetensors and SVD-ing a short-prompt cache) but skipped it: the checkpoint is a multi-GB download for a single illustrative spectrum, and the paper's measured tables plus this simulation already demonstrate the mechanism. Readers with the model cached locally can reproduce the paper's spectra with the authors' code⁸.

What to do with this on Monday

For anyone building or buying KV-cache compression, the paper converts into a short checklist:

1. Ask which axes the compressor truncates. Token/feature truncation is spectrally justified; head/layer truncation at moderate error is fighting a full-rank mode. If a vendor's "free 2x" mixes heads or layers, ask for the per-mode spectra.
2. Measure your own spectra before tuning anything — the mode-pinning certificate is checkable from one SVD per mode unfolding, and per-model quirks (GQA head counts, layer grouping) shift the constants, not the dichotomy. This is the same measure-before-you-tune discipline as prefix-cache working-set sizing (see also the KV-cache glossary).
3. Budget keys and values separately. Values need roughly twice the token-mode rank of keys at the same error; a shared rank wastes one side or starves the other⁴.
4. Compress keys pre-RoPE — 41%–64% of key compressibility evaporates after the rotation⁴.
5. Remember the memory economics. KV-cache compression is not an aesthetic exercise; cache bytes are the binding constraint on batch size and thus on serving cost — the same HBM-shortage arithmetic that makes a 2x cache reduction worth fighting for in the first place.

The deep irony in this paper is pleasant: the math that kills the "compress everything" hype also tells you exactly where compression genuinely works — along tokens and features, keys pre-RoPE, values in Tucker form, heads and layers left alone certified by a theorem. That is more useful than any vendor benchmark, because unlike a benchmark, the spectra tell you what the next model will do too, and they were cheap to measure all along.