Place the vocabulary on a request timeline
Prefix reuse, prefill, decode and latency metrics describe different points in the same serving loop.
AI-assisted learning visualization · schematic article explanation
A prefix is a shared beginning of a request. A matching cache entry can reuse earlier KV; the remaining uncached input still needs processing, and a hit does not retrieve a prewritten answer.
KV cache is one important part of LLM serving capacity and latency. This glossary gives definitions and arithmetic examples for common cache and scheduling terms. Numerical examples use the stated model configurations and assumptions from the KV-cache deep dive, the agent KV-tiering guide, the HBM4 economics analysis, and the UNISON scheduler teardown.
The terms
KV cache
The stored attention key (K) and value (V) states that let a decode step reuse projections from past tokens. The step still attends to relevant past positions. For conventional full attention, stored bytes grow approximately linearly with context length and active sequences. KV capacity can limit concurrency alongside weights, activations, compute and runtime allocations; hybrid architectures may store additional state.
Token
A unit in the model input/output tokenization. Text chunks and multimodal inputs are encoded differently across models; one image patch need not map one-to-one to a decoder token. Every quantity in this glossary is denominated in tokens — cache bytes, context length, throughput — so confusion between token counts and character counts breaks capacity math silently. Reference numbers here assume the site-standard per-token KV byte costs (GQA-8 fp16: 262,144 B; MLA bf16: 70,272 B).
Bytes-per-token formula
For conventional MHA/MQA/GQA caches that store separate K and V tensors: multiply attention layers × KV heads × head dimension × bytes per value × 2. On a 32B-class GQA model (64 layers, 8 KV heads, 128 head dim, fp16) it yields 256 KiB per token; at 32k context one user therefore holds 8.59 GB of cache. MLA and hybrid architectures need different formulas.
KV bytes = 2 × L × H_KV × d_head × b_dtype × s (s = tokens; the same per-token accounting as §3 of the PagedAttention paper)
Prefill
The phase in which the engine ingests the prompt: it processes input tokens in parallel and writes K/V states for attention layers. Prefill contributes to time-to-first-token; prefix caching can avoid repeated work. The 2 × active parameters × tokens approximation gives about 148 TFLOP, not 303 TFLOP, for 2,000 tokens on a 37B-active MoE. Attention and routing add work, while actual latency also depends on batch, kernels and memory traffic.
See: Agentic KV tiering
Decode
The phase in which the model generates output token by token. For a dense model at small batch, weight and KV reads can bound decode. At 8k context, the reference Qwen3-VL-32B's idealized bandwidth floor on an H200 SXM is about 14.35 ms per step for one user (~69.7 tok/s ceiling). Sixteen users fit in raw H200 capacity; sharing the weight read raises the aggregate bandwidth ceiling to ~760 tok/s. These are model ceilings, not measured rates.
MHA (multi-head attention)
The original transformer arrangement: every query head owns its own K/V pair, so KV pairs per token equal the query-head count. For otherwise equal head dimensions and layers, it uses more KV memory than MQA or GQA — at the reference config (64 heads, 128 dim, 64 layers, fp16) MHA stores 2,097,152 B = 2 MiB per token, i.e. 209.7 GB of cache per 100k-context session.
MQA (multi-query attention)
The aggressive sharing extreme: one single K/V head shared across all query heads ("the different heads share a single set of keys and values," Shazeer 2019), cutting the cache 64× at the reference config — 32 KiB (32,768 B) per token. Quality degrades on some tasks, which is why GQA exists as the interpolation.
GQA (grouped-query attention)
The middle design: one shared K/V head per small group of query heads. The illustrated 8-KV-head / 64-query-head shape cuts the per-token cache 8× versus otherwise identical MHA, to 262,144 B (256 KiB) at fp16. The GQA paper reports quality close to MHA at speed close to MQA and an uptraining recipe using about 5% of original pretraining compute. The 3.73× ratio to the DeepSeek-V3 MLA example below compares two different model configurations; it is not a controlled GQA-versus-MLA saving for the same model.
MLA (multi-head latent attention)
DeepSeek's mechanism: instead of storing K and V per head, store a compressed latent vector plus a decoupled RoPE key, and reconstruct K/V during attention. DeepSeek-V3's config keeps 576 elements per token per layer over 61 layers, bf16 → 70,272 B ≈ 68.6 KiB per token. The V2 paper reports a 93.3% smaller cache and 5.76× higher maximum generation throughput in its own comparison with a 67B MHA baseline. The 3.73× difference from the 256 KiB GQA example above includes differences in layer count and dimensions.
PagedAttention
vLLM's borrow from OS virtual memory: split each sequence's logical KV cache into fixed-size blocks, map them through a block table to scattered physical blocks, and allocate/free block-by-block on demand. The paper's measurement is the whole argument — reservation-style allocators used only 20.4–38.2% of KV memory for actual token states (the rest was reservation slack and fragmentation), while the paged allocator reached 96.3%.
Block
The fixed-granularity allocation unit of a paged KV cache — vLLM's default is 16 tokens. A 32,768-token sequence occupies 2,048 blocks; at the GQA-8 fp16 reference config each block holds 16 × 262,144 B = 4 MiB of K/V. The point of block granularity: a request that stops at 1,000 tokens pays for 63 blocks, not the 8.59 GB a full 32k contiguous reservation would lock away; waste shrinks to at most one partially-filled block per request.
Block table
The per-sequence mapping from logical token positions to physical block locations — the page table of the KV cache. It is what lets the cache be non-contiguous: attention gathers K/V rows through the table, physical blocks are shared between sequences where prefixes match, and reference counting decides when a block can actually be freed. Freed-block bookkeeping at block granularity is also the hook every eviction policy (LRU, TTL, survival-penalty) hangs off.
Radix tree / prefix caching
Organizing all cached KV in a tree keyed by token sequences, so any new request automatically reuses the longest matching cached prefix subject to the runtime's prefix matching and cache policy. SGLang's RadixAttention keeps finished-and-running KV in an LRU-managed radix tree; the paper reports up to 6.4× higher throughput on prefix-heavy workloads. For example, a 2,000-token shared system prompt occupies ~524 MB of GQA FP16 KV without reuse; a compatible prefix cache may share its stored blocks across requests.
See: vLLM vs SGLang
Prefix cache hit rate
The fraction of requested prefix tokens served from cache instead of prefill compute and KV writes. It is the metric that caps what caching can buy: UNISON's scheduler traces span hit-rate gains of just +0.3% (no contention — nothing to win) to +23.1% (baseline losing a quarter of the pool), and SGLang's cache-aware router moved hit rate from 20% to 75% on a shared-prefix workload by routing accordingly. At 0% hits, vLLM's V1 prefix cache still costs under 1% throughput — which is why it ships default-on.
See: UNISON scheduler analysis
Batch
The set of sequences whose decode steps execute together in one forward pass. The dense weight read is shared across the batch, while each sequence adds its own KV read. For the reference model at 8k context on H200, the idealized aggregate bandwidth ceiling rises from ~69.7 tok/s at batch 1 to ~760 tok/s at batch 16, while the per-user latency floor rises from 14.35 to 21.06 ms. Compute, KV capacity, runtime memory, and latency targets all constrain useful batch size.
Continuous batching
The scheduler design (Orca-lineage, default in vLLM/SGLang/TensorRT-LLM) that admits new requests and retires finished ones at every decode step, instead of waiting for a whole static batch to drain. It can improve utilization when request lengths differ and lets a scheduler admit requests between decode steps. Capacity and latency policies still govern admission.
See: vLLM vs SGLang
TTFT (time to first token)
Latency from request arrival to the first generated token, including queueing, prefill of uncached tokens and the first decode step. The dominant part depends on the workload. It is one metric prefix caching can improve: a hit avoids re-prefilling an entire history outright, which is a saved read-and-recompute of a ~7 GB (100k-token, MLA) prefix — UNISON reports TTFT reductions of 58% to 89% on long-horizon traces, and the reason the savings are enormous is exactly that the counterfactual is re-prefill.
See: UNISON scheduler analysis
TPOT (time per output token)
The time between generated tokens after the first. In a bandwidth-limited approximation, KV-read time grows with context: on an H200 SXM the reference model's 32k cache alone takes at least 1.79 ms to read per step. TPOT also depends on attention compute, batching, cache reuse, kernels, and scheduling; it need not grow linearly across every architecture or workload.
AMAT (average memory access time)
The memory-hierarchy average of a tier's hit latency and miss latency weighted by hit rate — the metric that prices tier placement. Formally AMAT = t_hot + (1 − hit_rate) × t_cold_penalty: every byte served from the cold tier drags the average by the full tier gap, which for a PCIe-attached host tier against HBM3e is ~76× in bandwidth (63 GB/s vs 4.8 TB/s) and 0.11–1.1 s in per-turn restore latency. UNISON reports AMAT reductions of 22% to 51% from better placement alone.
See: Agentic KV tiering
LRU eviction
Least-recently-used: evict the block whose KV was last touched longest ago. Access history is cheap to track, so LRU is a common baseline. A session waiting for a tool response can appear old and lose its reusable KV even if it will soon resume. Whether another policy beats LRU depends on the workload, cache size and evaluation metric.
See: UNISON scheduler analysis
TTL eviction
Time-to-live: expire KV after a fixed idle window. A short TTL reclaims memory sooner but can evict a session still waiting for a slow tool response. Its hit-rate and latency trade-off must be measured against the actual tool-return distribution.
See: UNISON scheduler analysis
Bélády MIN
The theoretical optimal eviction policy for a known future access trace: evict the block whose next use is farthest in the future. It is an offline benchmark, not an implementable policy for unknown requests. Comparing a scheduler with this benchmark shows the remaining gap under the measured trace; that gap alone does not prove which future events can or cannot be predicted. Learned, "relaxed-Belady" policies use estimates of future reuse.
See: UNISON scheduler analysis
Survival-penalty eviction
UNISON's SPEAR proposal fits a survival function to past tool-return gaps and penalizes eviction of sessions estimated to return soon. Past returns inform a probability, not a guarantee. The policy logic is a software design; integrating it with a particular serving engine would still require implementation and evaluation.
See: UNISON scheduler analysis
KV tiering
Storing the cache across a hierarchy — HBM hot tier, host DRAM/CXL/remote cold pool — so sessions that do not fit VRAM stay resident somewhere cheaper instead of being dropped. It converts capacity into a latency problem: re-reads from a PCIe 5.0 x16 host tier at ~63 GB/s take 0.112 s per 100k-token MLA turn and 1.12 s per 1M restore, versus 1.46 ms at HBM3e speed. TensorRT-LLM's KV Cache Manager V2 (hot HBM pool + cold pool with per-layer-group quotas and cold-page codecs) is the product form; placement policy, not tier capacity, is what matters.
See: Agentic KV tiering
Swap vs recompute
The two ways to honor a cache miss in a tiered system: swap restores the evicted KV bytes over the fabric (pay bandwidth and the tier's latency), recompute re-runs prefill over the affected tokens (pay FLOPs and re-read the source tokens). For a long prefix, swapping may be cheaper than recomputing; for a short prefix, recompute may be cheaper. The choice depends on transfer bandwidth and latency, compute availability, cache format and prefix length. The choice sets the AMAT penalty of every eviction.
See: UNISON scheduler analysis
Burst buffer
The HPC-land pattern now appearing in LLM serving: a fast intermediate staging layer that absorbs load spikes between sources and consumers — in agent serving, a storage/decode DMA path that re-units evicted KV with the decode GPUs without serializing through the prefill side. DualPath (arXiv:2602.21548) is the pattern's KV instantiation: it adds a storage-to-decode path via RDMA over the compute network and reports up to 1.87× offline and 1.96× online throughput over its in-house baseline by exactly this decoupling.
See: Agentic KV tiering
Idle-window migration
Using a session's own wait time as the DMA budget: while an agent is paused on a tool call, a scheduler may move KV between tiers before it resumes. The transfer still consumes bandwidth and power, but may avoid adding latency to the resumed request. This is UNISON's TIDE mechanism, and the timing is the whole idea — a 7 GB transfer may complete within a 10 s tool gap, whereas the same transfer on resume can add delay. The timing can materially affect latency.
See: UNISON scheduler analysis
Speculative decoding
Generate token candidates cheaply (with a small draft model, an EAGLE head, MTP, or n-gram lookup), then verify the draft in a target-model forward pass. With a correct rejection-sampling verifier, the target model's sampling distribution is preserved; other verification schemes need separate guarantees. The method trades draft and verification work for fewer serial target-model decode steps when enough draft tokens are accepted. SGLang's EAGLE-3 runs report 2.36× throughput on Llama 3.1 8B (158.34 → 373.25 tok/s on one H100; benchmark conditions in vLLM vs SGLang).
See: vLLM vs SGLang
Draft acceptance rate
The fraction of proposed draft tokens the target model actually keeps — the meter that decides whether speculation pays. Every extra accepted token per verify pass is one fewer full memory-bound decode step, but throughput also depends on draft and verification costs; a poor acceptance rate instead pays draft compute plus rejected-rollback overhead. Draft quality and cost both affect the result.
See: Groq LPX decode SRAM
KV quantization (FP8/FP4)
Storing the cache in a narrower data type halves (FP8) or quarters (FP4 payload) the bytes per value before scale metadata. vLLM exposes kv_cache_dtype="fp8"; FP8 KV approximately doubles tokens per fixed KV budget. At the 256 KiB/token GQA example, 100k tokens occupy 26.2 GB in bf16 or 13.1 GB in fp8. If 153.6 GB of a 192 GB card were available for KV after weights and runtime, the raw capacity would be 5.8 or 11.7 such sessions respectively; this is a hypothetical budget, not a universal 192 GB card result. FP4 cache support and quality depend on the serving engine and checkpoint.
Per-head / per-tensor scales
The granularity at which KV-quantization scales are chosen: per-tensor (one scale factor for the whole cache — simplest, coarsest) or per-attention-head (q_scale = [num_heads], k/v_scale = [num_kv_heads] in vLLM), which bounds outliers at head granularity and survives narrower data types. Per-channel/per-block refinements (KIVI-style) can help preserve quality with INT4-class KV; calibration options run from scales = 1.0 to dataset calibration via llm-compressor.
Working set (fleet)
The bytes of KV the running fleet needs resident to make progress: per-agent bytes/token × context × fleet size. It is the number that sizes everything else — 50 agents parked at 1M context on MLA are 50 × 70.27 GB = 3,514 GB of managed state, ~25 × 141 GB cards before a single weight is placed; 100 paused 100k agents are 702.72 GB of pure cache doing nothing. When the working set exceeds the hot tier, eviction (and its costs) begin; every policy in this glossary is an answer to that one overflow.
See: Agentic KV tiering
Prefill/decode disaggregation
Splitting prefill and decode onto different accelerators or node roles so compute-heavy prefill does not stall memory-bound decode in a shared batch. The 2026 wrinkle from the KV side: disaggregation externalizes cache placement — a turn's KV must move from wherever prefill wrote it to the decode replica, which is a fabric-and-burst-buffer problem (DualPath's storage-to-decode path is the response), and TensorRT-LLM-style hot/cold pool managers are the control plane that makes the hot tier's placement policy explicit.
See: Agentic KV tiering
Primary sources
- vLLM / PagedAttention paper
- Grouped-Query Attention paper
- DeepSeek-V2 MLA paper
- SGLang / RadixAttention paper
- UNISON preprint
- DualPath preprint
- Speculative decoding paper
How to use this glossary
Each entry is deliberately one paragraph: the definition plus the number that makes it real. If a term needs the full derivation, the linked article carries it — the deep dive for architecture and allocator math, the tiering guide for the bytes-moved view of agent fleets, the HBM4 piece for why all of this is procurement, and the UNISON piece for where eviction policy is heading. The through-line: every term above is a lever on the same two scarce quantities, KV bytes and the bandwidth to move them.