Skip to content
Friendly disclaimer: flozi00 TechHub is a solo side-project next to a full-time job — personal learning notes, no official statements. Verify critical steps yourself.
flozi00 Logoflozi00 TechHub
ENDE

KV Cache Memory Growth Explorer

Watch one agent's KV cache fill a node's free HBM as context grows — across GQA, MLA and hybrid attention models, with a live console fill bar per model.

2 min readflozi00
aigpuinferencevllmdeep-learningtoolscalculator
Skip to the full article
Watch · explore · understand

Explain the slope before reading the fill bar

The explorer combines a pooled memory budget with an architecture-specific per-token cache rate. Both assumptions shape its ceiling.

AI-assisted learning visualization · schematic article explanation

Step 1 / 4Read the architecture-specific slope

GQA stores full K/V for its KV groups; MLA stores a latent representation and positional state. A hybrid model may grow KV only in its attention layers while retaining separate recurrent state elsewhere.

ComparisonCurrent stepSelect a block to explain it
GQA, MLA or hybrid state
Each family stores a different attention representation.
Cache bytes added per token
Use actual cached layers and dimensions, not the family name alone.
  1. GQA, MLA or hybrid state Cache bytes added per token
    architecture-specific formula
What to remember

A steeper bar means more modeled cache bytes per token; it does not by itself imply a slower or worse model.

These are simplified pooled-capacity estimates. Hybrid fixed state, actual per-device placement and serving performance require separate accounting and measurement.

Terms and the complete walkthrough
Per-token slope
The modeled amount of additional cache storage for each retained token.
Latent KV
A compressed internal representation used instead of storing every expanded head vector.
Pooled capacity
An aggregate memory budget whose usability depends on actual placement across devices.
  1. Read the architecture-specific slope

    GQA stores full K/V for its KV groups; MLA stores a latent representation and positional state. A hybrid model may grow KV only in its attention layers while retaining separate recurrent state elsewhere.

  2. Change context without changing the weights

    Moving the context slider multiplies the modeled per-token rate by more retained tokens. The model weights remain the same size; changing their precision is a separate control that changes the free pool.

  3. Interpret the quotient as an aggregate limit

    The explorer adds group memory, subtracts its included allocations and divides by the estimated KV per context. This gives a capacity ceiling for its pooling assumptions, not a prediction of achievable concurrent throughput.

  4. Add excluded state before deployment

    The simplified curves omit activation peaks, recurrent states, fragmentation and other listed buffers. A total fit across GPUs also requires a runtime and topology that can place each device’s shard successfully.

All connections

  • GPU-group memory → Estimated free KV capacity: pooled total
  • Weight footprint and reserves → Estimated free KV capacity: subtract included state
  • GQA, MLA or hybrid state → Cache bytes added per token: architecture-specific formula
  • Cache bytes added per token → One context’s estimated KV: multiply by context length
  • Estimated free KV capacity → Aggregate capacity ceiling: available bytes
  • One context’s estimated KV → Aggregate capacity ceiling: bytes per context
Set your own pace. Pause at any time or select a step directly.

A GPU group has a fixed amount of device memory. Weights take their share, and some of the remainder can hold KV cache for concurrent requests. KV size depends on the attention architecture, the number of cached layers, sequence length and cache dtype.

Pick a GPU and group width, choose weight precision, then drag the context slider to compare GQA, MLA and hybrid attention. Each row assumes one model per GPU group with sharded weights and pooled memory. Pooling across devices requires a suitable parallel runtime and topology; an aggregate fit is not a per-device placement guarantee.

The curves are simplified capacity estimates, using the same model and GPU presets and kvPerToken() formula as the inference calculator. Their free-memory estimates can differ because the inference calculator also reserves activation memory and uses a different placement assumption.

Interactive Estimator

KV cache memory growth explorer

Watch one agent’s KV cache fill the node’s free HBM as its context grows — across attention families (GQA, MLA, hybrid linear attention). The same model and GPU data as the inference calculator, so every number stays consistent.

GLM-5.2 (753B-A40B, MLA + DSA)MoE + MLA + DeepSeek-style Sparse AttentionMLA ≈ ×43 smaller than ungrouped KV heads
87.8 KiB/token
KV at this context: 10.97 GiBConcurrent agents at this context: 116Max context for 1 agent: 1,048,576 tokens
DeepSeek V3.2-671B-A37B (MoE + MLA)MoE + Multi-head Latent AttentionMLA ≈ ×57 smaller than ungrouped KV heads
68.6 KiB/token
KV at this context: 8.58 GiBConcurrent agents at this context: 158Max context for 1 agent: 163,840 tokens
Qwen3-Next-80B-A3B (hybrid MoE)Hybrid Gated-DeltaNet + MoE
24.0 KiB/token
KV at this context: 3.00 GiBConcurrent agents at this context: 641Max context for 1 agent: 262,144 tokens
Llama 3.3-70B (dense)Dense GQA
320.0 KiB/token
KV at this context: 40.00 GiBConcurrent agents at this context: 48Max context for 1 agent: 131,072 tokens
Qwen3-32B (dense)Dense GQA
256.0 KiB/token
KV at this context: 32.00 GiB(exceeds model max context)Concurrent agents at this context: 0Max context for 1 agent: 40,960 tokens

Models

Qwen dense

Qwen MoE

Qwen3.8 / Qwen4-exp

Llama / open

DeepSeek

GLM / Z.ai (Zhipu)

Qwen3.5 / 3.6

NVIDIA

KV bytes per token = 2 · KV heads · head dim · precision · KV layers (GQA) or (latent + rope dim) · precision · KV layers (MLA); hybrid models only count full-attention layers. MTP draft layers and activation memory are excluded. Vocabulary: formula follows the inference calculator’s kvPerToken().

Why the families diverge so hard

Three architectural choices decide the slope of each curve:

  • GQA stores the full K and V for its KV groups. Llama-3.3-70B keeps 8 KV heads × 128 head-dim over 80 layers — 320 KiB per token in BF16.
  • MLA stores a compressed latent instead. In the GLM-5.2 preset, kv_lora_rank 512 + 64 RoPE dimensions yields ~88 KiB/token in BF16. A hypothetical ungrouped 64-KV-head baseline at the same head dimension would use ~3.7 MiB/token, about 43× more. This comparison is not an actual GQA variant of GLM-5.2.
  • Hybrid linear attention (Qwen3-Next) keeps attention KV in every fourth layer (2 KV heads × 256 head dimensions): ~24 KiB/token in BF16. The estimate excludes the fixed-size recurrent states of its other 36 layers.

In the default 8×B300 scenario with the GLM-5.2 preset and FP8 weight assumption, a 128 Ki-token request uses about 11 GiB of BF16 KV; the aggregate pool estimate admits 116 such requests. At 1 Mi tokens, it uses about 88 GiB and the estimate admits 14. These are arithmetic ceilings, not achievable concurrency or throughput predictions.

What the explorer deliberately excludes

The estimate ignores prefill activation peaks, prefix sharing, speculative-decoding KV, recurrent states, attention-index buffers, offloading, fragmentation and per-device placement. Some features reduce memory use and others increase it. Treat the result as an aggregate arithmetic bound.

Sources

Where to go deeper