Skip to content
Friendly disclaimer: flozi00 TechHub is a solo side-project next to a full-time job — personal learning notes, no official statements. Verify critical steps yourself.
flozi00 Logoflozi00 TechHub
ENDE

KV Cache Memory Growth Explorer

Watch one agent's KV cache fill a node's free HBM as context grows — across GQA, MLA and hybrid attention models, with a live console fill bar per model.

2 min readflozi00
aigpuinferencevllmdeep-learningtoolscalculator

A node has a fixed amount of HBM. Weights take their share, and what is left is the KV pool — the memory your running agents rent for their context. Every attention head a model stores per token decides how large that rent becomes.

The explorer below plots exactly that race. Pick a GPU and node width, choose your weights precision, then drag the context slider and watch each attention family (GQA, MLA, hybrid linear attention) eat into the pool at a very different rate. Each row assumes one model per node: its weights are already resident, and its agents share what remains.

All numbers are theoretical ceilings from vendor specs and the same kvPerToken() math as the inference calculator — model and GPU constants come from the identical data source, so the two tools can never disagree.

Interactive Estimator

KV cache memory growth explorer

Watch one agent’s KV cache fill the node’s free HBM as its context grows — across attention families (GQA, MLA, hybrid linear attention). The same model and GPU data as the inference calculator, so every number stays consistent.

GLM-5.2 (753B-A40B, MLA + DSA)MoE + MLA + DeepSeek-style Sparse AttentionMLA ≈ ×43 smaller than same-heads GQA
87.8 KiB/token
KV at this context: 10.97 GiBConcurrent agents at this context: 116Max context for 1 agent: 1,048,576 tokens
DeepSeek V3.2-671B-A37B (MoE + MLA)MoE + Multi-head Latent AttentionMLA ≈ ×57 smaller than same-heads GQA
68.6 KiB/token
KV at this context: 8.58 GiBConcurrent agents at this context: 158Max context for 1 agent: 163,840 tokens
Qwen3-Next-80B-A3B (hybrid MoE)Hybrid Gated-DeltaNet + MoE
24.0 KiB/token
KV at this context: 3.00 GiBConcurrent agents at this context: 641Max context for 1 agent: 262,144 tokens
Llama 3.3-70B (dense)Dense GQA
320.0 KiB/token
KV at this context: 40.00 GiBConcurrent agents at this context: 48Max context for 1 agent: 131,072 tokens
Qwen3-32B (dense)Dense GQA
256.0 KiB/token
KV at this context: 32.00 GiBConcurrent agents at this context: 61Max context for 1 agent: 131,072 tokens

Models

Qwen dense

Qwen MoE

Qwen3.8 / Qwen4-exp

Llama / open

DeepSeek

GLM / Z.ai (Zhipu)

Qwen3.5 / 3.6

NVIDIA

KV bytes per token = 2 · KV heads · head dim · precision · KV layers (GQA) or (latent + rope dim) · precision · KV layers (MLA); hybrid models only count full-attention layers. MTP draft layers and activation memory are excluded. Vocabulary: formula follows the inference calculator’s kvPerToken().

Why the families diverge so hard

Three architectural choices decide the slope of each curve:

  • GQA stores the full K and V for its KV groups. Llama-3.3-70B keeps 8 KV heads × 128 head-dim over 80 layers — 320 KiB per token in BF16.
  • MLA stores a compressed latent instead. GLM-style kv_lora_rank 512 + 64 rope-dim compresses to ~88 KiB/token — a hypothetical same-heads GQA build of that model would need ~3.7 MiB per token (about 43× more), which no single node survives at long context.
  • Hybrid linear attention (Qwen3-Next) keeps KV only in every 4th layer (2 KV heads × 256 head-dim), the other 36 layers carry a tiny fixed-size recurrent state — ~24 KiB/token plus a small constant.

The practical takeaway for fleet planning: on an 8×B300 node with GLM-5.2 fp8 weights resident, a 128K-context agent costs about 11 GiB of pool (KV in BF16), leaving ~116 concurrent agents at that context (raw, before any runtime margin). At 1M context that same agent costs 88 GiB and the node fits ~14 of them — long context is not just a "quality" axis, it is the dominant capacity axis.

What the explorer deliberately excludes

To stay honest, the estimate ignores prefill activation spikes, RadixAttention-style prefix reuse, MTP draft-layer KV, and offloading — every one of those shifts real deployments, and every one of them punishes naive planning. Treat the numbers as a ceiling on capacity, not a promise of throughput.

Where to go deeper