Every LLM serving question reduces to arithmetic on one tensor: the key-value cache. This glossary defines every term in the serving stack the way an engineer needs it — one rigorous definition per term, plus the formula or architectural reason where it matters. Numbers are the site-verified ones from the KV-cache deep dive, the agent KV-tiering guide, the HBM4 economics analysis, and the UNISON scheduler teardown.
The terms
KV cache
The store of attention key (K) and value (V) tensors a serving engine keeps for every token of every active sequence, so that each decode step does not recompute attention over all of history. It is memoization with a linear memory price: bytes grow with every token seen, per sequence, for the life of the request. The per-token cost is fixed by the model's attention architecture and data type, and the total by concurrent context — which is why KV capacity, not FLOPs, caps how many users a GPU serves.
Token
The atomic unit an LLM processes and bills: a text chunk (or image patch) mapped to one row in the KV cache and one column of attention. Every quantity in this glossary is denominated in tokens — cache bytes, context length, throughput — so confusion between token counts and character counts breaks capacity math silently. Reference numbers here assume the site-standard per-token KV byte costs (GQA-8 fp16: 262,144 B; MLA bf16: 70,272 B).
Bytes-per-token formula
The single formula from which all capacity math derives: multiply layers × KV heads × head dimension × bytes per value × 2 (K and V stored separately). On a 32B-class GQA model (64 layers, 8 KV heads, 128 head dim, fp16) it yields 256 KiB per token; at 32k context one user therefore holds 8.59 GB of cache.
KV bytes = 2 × L × H_KV × d_head × b_dtype × s (s = tokens; the same per-token accounting as §3 of the PagedAttention paper)
Prefill
The phase in which the engine ingests the prompt: it processes all input tokens in parallel (large matrix multiplies, compute-bound), writing one K/V row per token per layer. Prefill sets time-to-first-token and initializes the cache; prefix caching exists to make repeated prefill unnecessary. An agent's incremental per-turn prefill of ~2k tokens is roughly 303 TFLOP on a 37B-active MoE — under half a second of H100 compute, which is why agents are usually not FLOP-limited.
See: Agentic KV tiering
Decode
The phase in which the model generates output one token at a time, each step attending over the entire cached history. Decode is memory-bound: every step must read the weights plus this sequence's whole KV cache from HBM and use each byte essentially once. At 8k context on an H100 SXM the arithmetic-intensity floor is ~20.0 ms per step — about 50 tok/s for one user — while batching 16 users amortizes the weights read into 10.8× aggregate throughput.
MHA (multi-head attention)
The original transformer arrangement: every query head owns its own K/V pair, so KV pairs per token equal the query-head count. It quality-matches the checkpoint's pretraining but is the memory worst case — at the reference config (64 heads, 128 dim, 64 layers, fp16) MHA stores 2,097,152 B = 2 MiB per token, i.e. 209.7 GB of cache per 100k-context session.
MQA (multi-query attention)
The aggressive sharing extreme: one single K/V head shared across all query heads ("the different heads share a single set of keys and values," Shazeer 2019), cutting the cache 64× at the reference config — 32 KiB (32,768 B) per token. Quality degrades on some tasks, which is why GQA exists as the interpolation.
GQA (grouped-query attention)
The middle design: one shared K/V head per small group of query heads. The common g=8 configuration — 8 KV heads feeding 64 query heads — cuts the per-token cache 8× versus MHA to 262,144 B (256 KiB) at fp16, while the GQA paper's headline is quality close to MHA at speed close to MQA, plus an "uptraining" recipe converting existing MHA checkpoints with about 5% of original pretraining compute. Picking GQA over MLA in a memory-rationed market is a 3.73× capacity self-penalty.
MLA (multi-head latent attention)
DeepSeek's mechanism: instead of storing K and V per head, store a rank-compressed latent vector plus a small decoupled RoPE key, and reconstruct full K/V on the fly during attention. DeepSeek-V3's config keeps 576 latent elements per token per layer over 61 layers, bf16 → 70,272 B ≈ 68.6 KiB per token; the V2 paper reports a 93.3% smaller cache and 5.76× higher max generation throughput than its 67B MHA baseline. Against GQA-8 it is a further 3.73× compression.
PagedAttention
vLLM's borrow from OS virtual memory: split each sequence's logical KV cache into fixed-size blocks, map them through a block table to scattered physical blocks, and allocate/free block-by-block on demand. The paper's measurement is the whole argument — reservation-style allocators used only 20.4–38.2% of KV memory for actual token states (the rest was reservation slack and fragmentation), while the paged allocator reached 96.3%.
Block
The fixed-granularity allocation unit of a paged KV cache — vLLM's default is 16 tokens. A 32,768-token sequence occupies 2,048 blocks; at the GQA-8 fp16 reference config each block holds 16 × 262,144 B = 4 MiB of K/V. The point of block granularity: a request that stops at 1,000 tokens pays for 63 blocks, not the 8.59 GB a full 32k contiguous reservation would lock away; waste shrinks to at most one partially-filled block per request.
Block table
The per-sequence mapping from logical token positions to physical block locations — the page table of the KV cache. It is what lets the cache be non-contiguous: attention gathers K/V rows through the table, physical blocks are shared between sequences where prefixes match, and reference counting decides when a block can actually be freed. Freed-block bookkeeping at block granularity is also the hook every eviction policy (LRU, TTL, survival-penalty) hangs off.
Radix tree / prefix caching
Organizing all cached KV in a tree keyed by token sequences, so any new request automatically reuses the longest matching cached prefix without configuration or hashing. SGLang's RadixAttention keeps finished-and-running KV in an LRU-managed radix tree; the paper reports up to 6.4× higher throughput on prefix-heavy workloads. The free-money example: a 2,000-token system prompt costs ~524 MB of GQA fp16 KV per concurrent conversation — with tree sharing, computed once and stored once.
See: vLLM vs SGLang
Prefix cache hit rate
The fraction of requested prefix tokens served from cache instead of prefill compute and KV writes. It is the metric that caps what caching can buy: UNISON's scheduler traces span hit-rate gains of just +0.3% (no contention — nothing to win) to +23.1% (baseline losing a quarter of the pool), and SGLang's cache-aware router moved hit rate from 20% to 75% on a shared-prefix workload by routing accordingly. At 0% hits, vLLM's V1 prefix cache still costs under 1% throughput — which is why it ships default-on.
See: UNISON scheduler analysis
Batch
The set of sequences whose decode steps execute together in one forward pass. Batching is the economics of serving: the weights read is shared across the batch while each sequence's KV read is private, so batch 16 at 8k context on an H100 turns one weights read into 539 tok/s aggregate — 10.8× a single user's 49.9 tok/s — while each user waits ~48% longer per token. Capacity-wise, batch size is capped not by FLOPs but by KV bytes per user.
Continuous batching
The scheduler design (Orca-lineage, default in vLLM/SGLang/TensorRT-LLM) that admits new requests and retires finished ones at every decode step, instead of waiting for a whole static batch to drain. Without it, throughput collapses to bursty per-batch averages and prefix-cache hits go stale between batches; with it, the running batch is re-formed each step from everyone whose KV still fits — which makes the KV budget, again, the admission controller.
See: vLLM vs SGLang
TTFT (time to first token)
Latency from request arrival to the first generated token — dominated by prefill compute for the uncached portion of the prompt. It is the metric prefix caching pays off in: a hit avoids re-prefilling an entire history outright, which is a saved read-and-recompute of a ~7 GB (100k-token, MLA) prefix — UNISON reports TTFT reductions of 58% to 89% on long-horizon traces, and the reason the savings are enormous is exactly that the counterfactual is re-prefill.
See: UNISON scheduler analysis
TPOT (time per output token)
The latency of each decode step after the first token — set by how long one memory-bound step takes: read weights plus this sequence's full KV from HBM, emit one token. TPOT grows linearly with context (at 32k, batch 1, the cache read alone costs 2.6 ms every step on the reference config) and worsens with batching (~48% longer per token at batch 16). Perceived speed is TTFT once, TPOT every token.
AMAT (average memory access time)
The memory-hierarchy average of a tier's hit latency and miss latency weighted by hit rate — the metric that prices tier placement. Formally AMAT = t_hot + (1 − hit_rate) × t_cold_penalty: every byte served from the cold tier drags the average by the full tier gap, which for a PCIe-attached host tier against HBM3e is ~76× in bandwidth (63 GB/s vs 4.8 TB/s) and 0.11–1.1 s in per-turn restore latency. UNISON reports AMAT reductions of 22% to 51% from better placement alone.
See: Agentic KV tiering
LRU eviction
Least-recently-used: evict the block whose KV was last touched longest ago. It is the default proxy because access history is cheap to keep, and it fails on agents precisely because a tool-waiting session has not touched its KV since its last turn — it looks older than a chatty user and is evicted first, destroying exactly the sessions with the most banked context. LRU is the baseline every 2026 paper beats and the null hypothesis every policy claim must be measured against.
See: UNISON scheduler analysis
TTL eviction
Time-to-live: expire KV after a fixed idle window regardless of access history. The agent trap is heavy-tailed tool latency — any TTL short enough to reclaim memory also kills the sessions whose tool calls happen to be slow (a browser or sandbox step can take tens of seconds). LRU fails by misranking; TTL fails on schedule: both ask the access log a question only the job structure (a tool call implies a return) can answer.
See: UNISON scheduler analysis
Bélády MIN
The theoretical optimal eviction policy: evict the block whose next use is farthest in the future — computable only with an oracle. It is the ceiling every real policy is measured against; the useful framing from UNISON's evaluation is that a good scheduler closes most of the distance from LRU toward Bélády-class foresight, and the remaining oracle gap tells you how much prediction the workload genuinely forbids (some tool outcomes are unpredictable, and no hazard model recovers that). Learned, "relaxed-Belady" policies are the same idea with predicted futures.
See: UNISON scheduler analysis
Survival-penalty eviction
The 2026 refinement: fit a survival function over each session's tool-return gap (gap average plus a turn-indexed hazard — a session that returned from 40 tool calls almost certainly returns from its 41st) and penalize eviction of sessions likely to come back. This is UNISON's SPEAR policy, the same statistical shift that took CDN caching from LRU to learned policies, and — the honest caveat from the paper — its ranking half is software-replicable today; a runtime could implement it as a patch to its block manager without new silicon.
See: UNISON scheduler analysis
KV tiering
Storing the cache across a hierarchy — HBM hot tier, host DRAM/CXL/remote cold pool — so sessions that do not fit VRAM stay resident somewhere cheaper instead of being dropped. It converts capacity into a latency problem: re-reads from a PCIe 5.0 x16 host tier at ~63 GB/s take 0.112 s per 100k-token MLA turn and 1.12 s per 1M restore, versus 1.46 ms at HBM3e speed. TensorRT-LLM's KV Cache Manager V2 (hot HBM pool + cold pool with per-layer-group quotas and cold-page codecs) is the product form; placement policy, not tier capacity, is what matters.
See: Agentic KV tiering
Swap vs recompute
The two ways to honor a cache miss in a tiered system: swap restores the evicted KV bytes over the fabric (pay bandwidth and the tier's latency), recompute re-runs prefill over the affected tokens (pay FLOPs and re-read the source tokens). For long agent prefixes swap wins almost always — re-prefilling a ~7 GB, 100k-token prefix costs the entire read-and-recompute, which is the worst case the UNISON tiering analysis prices — while for short blocks recompute can be cheaper than fabric round-trips. The choice sets the AMAT penalty of every eviction.
See: UNISON scheduler analysis
Burst buffer
The HPC-land pattern now appearing in LLM serving: a fast intermediate staging layer that absorbs load spikes between sources and consumers — in agent serving, a storage/decode DMA path that re-units evicted KV with the decode GPUs without serializing through the prefill side. DualPath (arXiv:2602.21548) is the pattern's KV instantiation: it adds a storage-to-decode path via RDMA over the compute network and reports up to 1.87× offline and 1.96× online throughput over its in-house baseline by exactly this decoupling.
See: Agentic KV tiering
Idle-window migration
Using a session's own wait time as the DMA budget: while an agent is paused on a tool call, memory bandwidth is idle, and moving its KV between tiers then is free in a way that competing with the resume prefill is not. This is UNISON's TIDE mechanism, and the timing is the whole idea — a 7 GB migration inside a 10 s tool gap costs nothing; the same migration started as the tool returns competes with resume and is a pure loss. Idle-window scheduling is the difference between tiering being viable and tiering being a latency bug.
See: UNISON scheduler analysis
Speculative decoding
Generate token candidates cheaply (with a small draft model, an EAGLE head, MTP, or n-gram lookup), then verify the whole draft in one parallel forward pass of the target model. It is algorithmically lossless — the verifier's distribution is preserved — and it attacks decode's serial memory-bound bottleneck by trading verify-pass FLOPs for accepted drafts per step. SGLang's EAGLE-3 runs show 2.36× throughput on Llama 3.1 8B (158.34 → 373.25 tok/s on one H100; figures as benchmarked in vLLM vs SGLang); one sharp edge: in vLLM, pipeline parallelism and speculative decoding are incompatible as of ≤ 0.15.0.
See: vLLM vs SGLang
Draft acceptance rate
The fraction of proposed draft tokens the target model actually keeps — the meter that decides whether speculation pays. Every extra accepted token per verify pass is one fewer full memory-bound decode step, so throughput scales with it; a poor acceptance rate instead pays draft compute plus rejected-rollback overhead. This is why draft quality dominates: an LPU/Groq-class SRAM-local drafter is attractive precisely because it can push tokens cheaply while the big card verifies in parallel.
See: Groq LPX decode SRAM
KV quantization (FP8/FP4)
Storing the cache in a narrower data type — halves (FP8) or quarters (FP4) the bytes per value while keeping architecture and allocator untouched. vLLM exposes kv_cache_dtype="fp8" (e4m3/e5m2, CUDA 11.8+) with the docs' capacity-first framing: FP8 KV approximately doubles the tokens that fit in the same budget — GQA-8 at 100k context drops from 26.2 GB to 13.1 GB per session, taking a 192 GB card from 5.8 to 11.6 concurrent agents. FP4-class cache pushes further at accuracy cost.
Per-head / per-tensor scales
The granularity at which KV-quantization scales are chosen: per-tensor (one scale factor for the whole cache — simplest, coarsest) or per-attention-head (q_scale = [num_heads], k/v_scale = [num_kv_heads] in vLLM), which bounds outliers at head granularity and survives narrower data types. Per-channel/per-block refinements (KIVI-style, per the HBM4 guide) are what make INT4-class KV viable at all; calibration options run from scales = 1.0 to dataset calibration via llm-compressor.
Working set (fleet)
The bytes of KV the running fleet needs resident to make progress: per-agent bytes/token × context × fleet size. It is the number that sizes everything else — 50 agents parked at 1M context on MLA are 50 × 70.27 GB = 3,514 GB of managed state, ~25 × 141 GB cards before a single weight is placed; 100 paused 100k agents are 702.72 GB of pure cache doing nothing. When the working set exceeds the hot tier, eviction (and its costs) begin; every policy in this glossary is an answer to that one overflow.
See: Agentic KV tiering
Prefill/decode disaggregation
Splitting prefill and decode onto different accelerators or node roles so compute-heavy prefill does not stall memory-bound decode in a shared batch. The 2026 wrinkle from the KV side: disaggregation externalizes cache placement — a turn's KV must move from wherever prefill wrote it to the decode replica, which is a fabric-and-burst-buffer problem (DualPath's storage-to-decode path is the response), and TensorRT-LLM-style hot/cold pool managers are the control plane that makes the hot tier's placement policy explicit.
See: Agentic KV tiering
How to use this glossary
Each entry is deliberately one paragraph: the definition plus the number that makes it real. If a term needs the full derivation, the linked article carries it — the deep dive for architecture and allocator math, the tiering guide for the bytes-moved view of agent fleets, the HBM4 piece for why all of this is procurement, and the UNISON piece for where eviction policy is heading. The through-line: every term above is a lever on the same two scarce quantities, KV bytes and the bandwidth to move them.