The KV Cache: Bit-Exact Memory Math, GQA vs MQA vs MLA, and PagedAttention

Every LLM serving question — how many users fit on one GPU, why long context dies, what FP8 KV actually buys — is answered by one tensor: the key-value cache. This guide derives its exact size at the byte level, then walks the three attack axes (architecture, allocator, data type) with real numbers for every claim.

The one formula

Each transformer layer stores one K and one V vector per token per KV head — recomputing all of history every step would be quadratic, so it is cached once and read every step:

• L = layers, H_KV = KV heads, d_head = head dimension, b_dtype = bytes per value, s = tokens
• Factor 2 = K and V stored separately
• Same formula as our VRAM calculator; matches Eq. 1 of the PagedAttention paper (arXiv:2309.06180)

Reference model across this guide (same site-verified config as the VRAM guide): Qwen3-VL-32B — 64 layers, 8 KV heads (GQA), 128 head dim.

Per token, fp16:

For comparison: the fp16 weights of this model are 66.7 GB (33,357,390,064 params × 2 bytes). So at max context a single user's KV cache (68.72 GB) is already larger than the entire model — the cache crossover sits at roughly 254,500 tokens. Context is not free; it is linear memory per user, every user, always.

Why it is also a bandwidth problem

Decode is memory-bound: one forward pass must read essentially the whole weights tensor plus this user's full KV cache from HBM, use each byte once, and produce a single token. On an H100 SXM (80 GB HBM3 at 3.35 TB/s per the datasheet), the arithmetic-intensity model gives a floor:

• Batch 1, 8k context: (65 + 2.15) GB / 3.35 TB/s ≈ 20.0 ms → 49.9 tok/s ceiling for one user
• Batch 16, 8k each: (65 + 34.4) GB / 3.35 TB/s ≈ 29.7 ms per step, but the weights read is shared → 33.7 tok/s per user, 539 tok/s aggregate — 10.8× more total throughput

That is the entire economics of batching: amortize the weights read over more sequences. Each user waits ~48% longer per token than solo (29.7 ms vs 20.0 ms), and the card produces nearly eleven times the tokens. (This linear-bandwidth model ignores attention FLOPs, cache hits, and kernel overhead — it is a ceiling, not a benchmark.)

At 32k context, batch 1: (65 + 8.59) GB per step → 45.5 tok/s — the cache read alone costs 2.6 ms every step, growing with every token generated.

Axis 1: Architecture — MHA vs MQA vs GQA vs MLA

The architecture decides how many K/V vectors each token costs. Four families, per-token cache at the reference config (fp16):

These byte numbers are what the architecture names mean. MQA shares one K/V head pair across all 64 query heads — in the paper's own phrase, "the different heads share a single set of keys and values" — cutting the cache 64×. GQA interpolates between the two: a fixed number of KV groups, here 8 of 64 query heads, hence an 8× cut; the paper's headline result is quality close to MHA at speed close to MQA, plus an "uptraining" recipe that converts existing MHA checkpoints with about 5% of the original pretraining compute. MLA is a different mechanism entirely: instead of storing K and V per head, it stores a rank-512 compressed latent vector plus a 64-dim decoupled RoPE key — 576 elements per token per layer in DeepSeek-V3's published config — and reconstructs full K/V on the fly during attention. The V2 paper reports a 93.3% smaller cache and 5.76× higher max generation throughput than its 67B MHA dense baseline.

For capacity planning you rarely choose the attention variant — you choose a model, and the variant comes with it. The useful takeaway is knowing what you bought per token, from the table above.

Axis 2: The allocator — contiguous reservation vs PagedAttention

The second axis is not about the model at all. A serving engine must place each request's growing (k,v) rows somewhere in VRAM. The naive scheme — reserve one contiguous buffer sized for the maximum context per request — wastes enormous amounts of memory: reservations are rarely filled, and contiguous chunks fragment as requests of different lengths come and go.

The PagedAttention paper (Kwon et al., SOSP 2023, arXiv:2309.06180) measured this: in then-standard systems, only 20.4–38.2% of the memory allocated for KV caches actually stored token states — the rest was reservation slack and fragmentation. vLLM's paged allocator, borrowing from OS virtual memory, reached 96.3% effective use in the same experiment: logical per-request caches are split into fixed 16-token blocks (vLLM's default block size), mapped through a per-sequence block table to scattered physical blocks, allocated on demand and freed token-block by token-block.

Block math (block size 16): a 32,768-token sequence occupies 2,048 blocks; at the reference config each block holds 16 × 262,144 B = 4 MiB of K/V. A request that stops at 1,000 tokens pays for 63 blocks (1,000/16 → 62.5 → 63 with the partial tail), not the 8.59 GB a full 32k reservation would lock away. Waste shrinks from "most of the buffer" to "at most one partially-filled block per request."

Prefix reuse — RadixAttention and friends

Paging fixes fragmentation; a second allocator-level win is sharing. Prompts are full of repeated prefixes — the same system prompt for every user, the same few-shot examples across a benchmark, the whole prior conversation in multi-turn chat. If each request recomputes and re-stores its own copy of that prefix KV, you pay compute and memory for the same bytes over and over.

SGLang's RadixAttention (arXiv:2312.07104) keeps finished-and-running KV in a radix tree keyed by token sequences, with LRU eviction and reference-counted nodes kept on the CPU. A new prompt that shares a prefix with anything in the tree reuses that branch's KV without recomputation: the common shares are found automatically, no manual prefix configuration. The paper reports up to 6.4× higher throughput on prefix-heavy workloads vs. systems without automatic reuse; the tree lookup itself adds negligible overhead.

Prefix math (reference config): a 2,000-token system prompt costs 2,000 × 262,144 B ≈ 524 MB of KV per conversation at fp16 — recomputed and re-stored for every concurrent user without prefix sharing. With tree sharing it is computed once and stored once. Multi-turn chat is the extreme case: turn n shares the entire prefix of turns 1…n−1, so the incremental KV cost of each turn is only its new tokens.

Axis 3: Data type — FP8 KV cache

The third axis halves the per-value byte cost: keep the architecture, keep the allocator, store the cache in FP8. vLLM exposes this as kv_cache_dtype="fp8" (choices: auto, fp8_e4m3, fp8_e5m2; CUDA 11.8+), with two quantization schemes — per-tensor scales, or per-attention-head scales (q_scale = [num_heads], k/v_scale = [num_kv_heads]). The docs' own framing is capacity-first: FP8 KV approximately doubles the number of tokens that fit in the same cache budget — supporting either longer contexts or more concurrent requests; with FlashAttention-3 the attention runs quantized end-to-end. Calibration approaches: no calibration (scales = 1.0), on-the-fly random-token calibration, or dataset calibration via llm-compressor (recommended).

Worked capacity example (H100 80 GB): budget = 80 × 0.9 = 72 GB. Weights fp16 = 66.7 GB → 5.3 GB left → fp16 KV: 20,162 tokens; fp8 KV: 40,323 tokens. Same GPU, quantized weights (AWQ/GPTQ ~18.7 GB) → free 53.3 GB → fp16 KV: 203,399 tokens; fp8 KV: 406,798 tokens. The reference model's max context is 262,144 tokens: fp16 weights cannot even hold one max-context user; quantized weights + fp8 KV holds it with room.

What to remember

• One formula: KV bytes = 2 × L × H_KV × d_head × b_dtype × s. Everything else is derived.
• Architecture is fixed with the model: GQA-8 → 256 KiB/token fp16; MQA → 32 KiB; MLA → ~70 KB (DeepSeek-V3). Know what you bought.
• Paging beats planning: naive contiguous reservation wastes up to ~80% of the KV budget (20.4–38.2% effective); PagedAttention-style paged allocators reach 96.3%.
• Prefix reuse is free money: 2,000-token system prompt ≈ 524 MB KV at the reference config — shared once via a radix tree across users, not recomputed per user.
• FP8 halves the per-token cost: from 256 KiB to 128 KiB — doubling capacity for the same budget, per the vLLM feature docs.
• KV crosses weights at ~248k tokens at the reference config (fp16): below that, weights dominate the budget; above it, cache dominates — and quantizing weights buys the most where cache dominates (both segments shown in the worked example above).
• Decoding is memory-bound: batch-1 decode reads the entire weights+this-user-KV per token: ~50 tok/s ceiling on H100 SXM at 8k ctx; batching 16 users turns one weights read into 10.8× aggregate throughput.