The KV Cache: Bit-Exact Memory Math, GQA vs MQA vs MLA, and PagedAttention

New to KV cache? Watch prefill, decode and cache reuse step by step before working through the memory formulas below.

Ripple-65B illustrates why a 1,024-position attention mask does not automatically cap stored K/V. Its guide derives an ideal compact hybrid payload separately from the unconfigured DynamicCache path in the released HF reference.

Kimi K3.1 Pro selects up to 2,048 positions for a global attention read but retains history for future queries. Its guide separates MLA payload, index keys and sparse read cost rather than treating top-k as cache eviction.

Every LLM serving question — how many users fit on one GPU, why long context dies, what FP8 KV actually buys — is answered by one tensor: the key-value cache. This guide derives its exact size at the byte level, then walks the three attack axes (architecture, allocator, data type) with real numbers for every claim.

The one formula

Each transformer layer stores one K and one V vector per token per KV head — recomputing all of history every step would be quadratic, so it is cached once and read every step:

• L = layers, H_KV = KV heads, d_head = head dimension, b_dtype = bytes per value, s = tokens
• Factor 2 = K and V stored separately
• Same formula as our VRAM calculator; matches Eq. 1 of the PagedAttention paper (arXiv:2309.06180)

For mixed local/global stacks, sum the retained positions separately by layer type. Mellum2.1 has 21 layers with a 1,024-position window and seven full-attention layers. Its guide derives the hybrid BF16 payload and distinguishes the window's logical working set from cache slices, temporary tensors and physical runtime allocations.

Reference model across this guide (same site-verified config as the VRAM guide): Qwen3-VL-32B — 64 layers, 8 KV heads (GQA), 128 head dim.

KV reuse is not limited to autoregressive decoding. Qwen-Image 2.1 makes text and reference-image states independent of the diffusion timestep and caches that fixed prefix across denoising steps. The same K/V byte formula applies, but the cached positions, reuse axis, and compute that remains are different.

Per token, fp16:

For comparison: the fp16 weights of this model are 66.7 GB (33,357,390,064 params × 2 bytes). So at max context a single user's KV cache (68.72 GB) is already larger than the entire model — the cache crossover sits at roughly 254,500 tokens. Context is not free; it is linear memory per user, every user, always.

Why it is also a bandwidth problem

At small batch, dense-model decode can be memory-bound: a step reads the weights plus each sequence's cache. On an H200 SXM (141 GB HBM3e at 4.8 TB/s ¹), both of the following configurations fit in raw capacity. The idealized bandwidth model gives a latency floor:

• Batch 1, 8k context: (66.7 + 2.15) GB / 4.8 TB/s ≈ 14.35 ms → 69.7 tok/s bandwidth ceiling for one user
• Batch 16, 8k each: (66.7 + 34.36) GB / 4.8 TB/s ≈ 21.06 ms per step, with the weights read shared → 47.5 tok/s per user, 760 tok/s aggregate — 10.9× the batch-1 aggregate ceiling

Batching amortizes the weight read over more sequences. Each user has ~47% longer idealized time per token than solo (21.06 ms vs 14.35 ms), while the aggregate bandwidth ceiling rises. Both cases fit in 141 GB before runtime reservations (batch 16: 66.7 + 34.36 ≈ 101.1 GB). This model omits attention FLOPs, memory traffic from other tensors, kernels and runtime reservations; it is not a benchmark or a guaranteed throughput.

At 32k context, batch 1: (66.7 + 8.59) GB / 4.8 TB/s → about 63.7 tok/s as an idealized bandwidth ceiling; reading the cache alone takes at least 1.79 ms per step.

Axis 1: Architecture — MHA vs MQA vs GQA vs MLA

The architecture decides how many K/V vectors each token costs. Four families, per-token cache at the reference config (fp16):

These byte numbers are what the architecture names mean. MQA shares one K/V head pair across all 64 query heads — in the paper's own phrase, "the different heads share a single set of keys and values" — cutting the cache 64×. GQA interpolates between the two: a fixed number of KV groups, here 8 of 64 query heads, hence an 8× cut; the paper's headline result is quality close to MHA at speed close to MQA, plus an "uptraining" recipe that converts existing MHA checkpoints with about 5% of the original pretraining compute. MLA is a different mechanism entirely: instead of storing K and V per head, it stores a rank-512 compressed latent vector plus a 64-dim decoupled RoPE key — 576 elements per token per layer in DeepSeek-V3's published config — and reconstructs full K/V on the fly during attention. The V2 paper reports a 93.3% smaller cache and 5.76× higher max generation throughput than its 67B MHA dense baseline.

For capacity planning you rarely choose the attention variant — you choose a model, and the variant comes with it. The useful takeaway is knowing what you bought per token, from the table above.

Axis 2: The allocator — contiguous reservation vs PagedAttention

The second axis is not about the model at all. A serving engine must place each request's growing (k,v) rows somewhere in VRAM. The naive scheme — reserve one contiguous buffer sized for the maximum context per request — wastes enormous amounts of memory: reservations are rarely filled, and contiguous chunks fragment as requests of different lengths come and go.

The PagedAttention paper (Kwon et al., SOSP 2023, arXiv:2309.06180) measured this: in then-standard systems, only 20.4–38.2% of the memory allocated for KV caches actually stored token states — the rest was reservation slack and fragmentation. vLLM's paged allocator, borrowing from OS virtual memory, reached 96.3% effective use in the same experiment: logical per-request caches are split into fixed 16-token blocks (vLLM's default block size), mapped through a per-sequence block table to scattered physical blocks, allocated on demand and freed token-block by token-block.

Block math (block size 16): a 32,768-token sequence occupies 2,048 blocks; at the reference config each block holds 16 × 262,144 B = 4 MiB of K/V. A request that stops at 1,000 tokens pays for 63 blocks (1,000/16 → 62.5 → 63 with the partial tail), not the 8.59 GB a full 32k reservation would lock away. Waste shrinks from "most of the buffer" to "at most one partially-filled block per request."

Prefix reuse — RadixAttention and friends

Paging fixes fragmentation; a second allocator-level win is sharing. Prompts are full of repeated prefixes — the same system prompt for every user, the same few-shot examples across a benchmark, the whole prior conversation in multi-turn chat. If each request recomputes and re-stores its own copy of that prefix KV, you pay compute and memory for the same bytes over and over.

SGLang's RadixAttention (arXiv:2312.07104) keeps finished-and-running KV in a radix tree keyed by token sequences, with LRU eviction and reference-counted nodes kept on the CPU. A new prompt that shares a prefix with anything in the tree reuses that branch's KV without recomputation: the common shares are found automatically, no manual prefix configuration. The paper reports up to 6.4× higher throughput on prefix-heavy workloads vs. systems without automatic reuse; the tree lookup itself adds negligible overhead.

Prefix math (reference config): a 2,000-token system prompt costs 2,000 × 262,144 B ≈ 524 MB of KV per conversation at fp16 — recomputed and re-stored for every concurrent user without prefix sharing. With tree sharing it is computed once and stored once. Multi-turn chat is the extreme case: turn n shares the entire prefix of turns 1…n−1, so the incremental KV cost of each turn is only its new tokens.

Axis 3: Data type — FP8 KV cache

For a model-specific architecture and serving example, the Kolibri 1 guide discusses local/global cache accounting alongside its FP8 serving path.

The third axis halves the per-value byte cost: keep the architecture, keep the allocator, store the cache in FP8. vLLM exposes this as kv_cache_dtype="fp8" (choices: auto, fp8_e4m3, fp8_e5m2; CUDA 11.8+), with two quantization schemes — per-tensor scales, or per-attention-head scales (q_scale = [num_heads], k/v_scale = [num_kv_heads]). The docs' own framing is capacity-first: FP8 KV approximately doubles the number of tokens that fit in the same cache budget — supporting either longer contexts or more concurrent requests; with FlashAttention-3 the attention runs quantized end-to-end. Calibration approaches: no calibration (scales = 1.0), on-the-fly random-token calibration, or dataset calibration via llm-compressor (recommended).

Worked capacity example (H100 80 GB): budget = 80 × 0.9 = 72 GB. Weights fp16 = 66.7 GB → 5.3 GB left → fp16 KV: 20,162 tokens; fp8 KV: 40,323 tokens. Same GPU, quantized weights (AWQ/GPTQ ~18.7 GB) → free 53.3 GB → fp16 KV: 203,399 tokens; fp8 KV: 406,798 tokens. The reference model's max context is 262,144 tokens: fp16 weights cannot even hold one max-context user; quantized weights + fp8 KV holds it with room.

What to remember

Sparse reads are not automatically compressed storage. Step-5-Preview searches eight-token proxy summaries but reads selected original K/V, retaining full history in its global layers. Its guide derives the cache payload, TP replication and standalone prefill workspace separately.

The cache layout also depends on implementation. Ling-3.1-flash illustrates the difference: its reference stores expanded MLA keys/values, while an absorbed serving path can retain the shared latent and positional key. The guide derives both budgets and includes the independent KDA state.

For a concrete hybrid exception to counting every layer as attention, LFM2.5-24B-A2B has ten GQA caches and 30 short-convolution states. Its configuration gives 20 KiB of BF16 KV per token plus 360 KiB of ordinary convolution state per sequence; the architecture guide separates this logical payload from resident expert weights and runtime allocations.

• One formula: KV bytes = 2 × L × H_KV × d_head × b_dtype × s. Everything else is derived.
• Architecture is fixed with the model: GQA-8 → 256 KiB/token fp16; MQA → 32 KiB; MLA → ~70 KB (DeepSeek-V3). Know what you bought.
• Paging beats planning: naive contiguous reservation wastes up to ~80% of the KV budget (20.4–38.2% effective); PagedAttention-style paged allocators reach 96.3%.
• Prefix reuse is free money: 2,000-token system prompt ≈ 524 MB KV at the reference config — shared once via a radix tree across users, not recomputed per user.
• FP8 halves the per-token cost: from 256 KiB to 128 KiB — doubling capacity for the same budget, per the vLLM feature docs.
• KV crosses weights at ~254,500 tokens at the reference config (fp16); compare the 66.7 GB weights with 262,144 bytes of KV per token.
• Bandwidth is one decode limit: on an H200 SXM at 8k context, the idealized ceilings are ~70 tok/s for one request and ~760 tok/s aggregate for 16. Measured rates depend on compute, runtime and scheduling.

Beam 1.0's hybrid cache grows with only nine global GQA layers; 43 other layers have a bounded 512-token attention window. Its guide derives the approximately 18.04 GiB FP8 KV payload at 1,048,576 tokens and explains why that is not a peak-memory or cache-quality measurement.

Visual inputs also consume sequence positions. Nemotron 3.5 Super VL combines paired-frame projection, spatial merging and change-based video pruning. Its guide derives the retained visual positions and their eight-block backbone KV contribution, while keeping encoder work, Mamba/MTP states and marker tokens separate.