Your Agent Fleet Is a Storage Array: The KV-Cache Tiering Math Behind Agentic Inference

Agent inference is sold as "the same GPUs, longer prompts." The arithmetic says otherwise. A multi-turn agent loop does not look like a chat workload at all: it looks like a storage array with GPUs attached — a fleet of processes whose dominant cost is not tokens-per-second but bytes of KV cache moved per turn, and whose capacity planning should be done in gigabytes of working set, not in requests or FLOPs. This guide does the per-turn, per-fleet math with real numbers, every figure computed from the site-verified KV-cache formula and primary sources.

The workload shape: long-prefix re-reads plus small increments

Chat is short prompts, few turns, small state. An agentic loop — plan, call a tool, read the result, plan again — does the opposite. The session context grows monotonically: every turn re-reads the whole prefix and appends a small increment. Per turn, an agent at 100k context with a 2,048-token tool result and 512 generated tokens does:

• Prefix re-read: the full KV of ~100k prior tokens must be available to attention at every decode step — re-read 512 times during the reply.
• Increment: only 2,048 + 512 tokens of new KV get computed and written.

Reference model: DeepSeek-V3 MLA¹ — the site-verified config (KV-cache guide): 576 latent elements per token per layer, 61 layers, bf16 → 70,272 B ≈ 68.6 KiB per token. Even after MLA's 93.3% compression against MHA, the multiplying factor is context length:

That is one agent residing on the GPU. Now count the bytes that must actually move.

Per-turn HBM bytes: agent vs chat

During decode, each step reads this worker's full KV from HBM (the weights read is shared across the batch; the KV read is not — it is per-sequence, every step, always). Computed directly:

An agent turn moves 25× the HBM bytes of a chat turn at the same model — and ratio scales linearly with context, because the read-per-step is the whole session, not the increment. The KV write is noise; the re-read is the workload. This one ratio is why agent token costs don't look like chat token costs: chat amortizes, agents re-scan.

Prefill vs decode framing: the incremental prefill of a turn (~2k tokens) is FLOPs-heavy — about 2 × 2 × 37B-active-params × 2,048 ≈ 303 TFLOP, less than half a second on one H100. The decode is bytes-heavy: 3,598 GB of pure re-reads per turn. Neither is the bottleneck shape of chat.

The bandwidth wall: 50 concurrent 1M-context workers

Scale to a modest agent fleet: 50 workers parked at 1M context each.

To decode at all, every worker re-reads its own full cache every step (partially from HBM, partially from wherever tiering put the rest). Aggregate read bandwidth just to keep the fleet decoding, computed at three per-worker rates:

A single H100 reads 3.35 TB/s. On ~25 cards (the KV-only capacity of a 141 GB-class fleet — an H200-class card, HBM3e at 4.8 TB/s — computed below), 70.3 TB/s means ~2.8 TB/s of pure KV traffic per card — 84% of an H100’s 3.35 TB/s, or 59% of that H200’s 4.8 TB/s, spent moving one worker's own history back to itself, with zero attention math done.

The GPU sits mostly idle waiting: for a 37B-active MoE token at 100k context, the arithmetic takes ~75 µs of SM time while the KV read takes ~2.1 ms — a compute utilization of 3.4%. The card is not FLOP-starved; it is storage-I/O bound in exactly the sense a database server is. This is the observation HotInfra'26² puts at the center: Reimagining LLM Inference Infrastructure with Memory-Centric KV Cache Servers (Kiyawat & Skadron, HotInfra '26, co-located with ISCA '26) proposes disaggregating KV into memory-centric servers on a CXL/PNM/PIM device spectrum — CPU-attached CXL DDRx expanders through near-memory compute nodes — precisely because the KV tier, not the GPU tier, is the resource the workload saturates. Their exemplar workload is a 32K-token generation run. The pattern has a name in their vocabulary: a KV cache server — storage-array thinking, formally proposed. The mechanics mirror what the NVLink bandwidth ground truth and our scale-up networking analysis show elsewhere: the fabric the workload actually needs is the memory fabric, and GPUs hang off it like accelerators.

The tiering pattern

DualPath (arXiv:2602.21548) states the problem in its own first sentence: "The performance of multi-turn, agentic LLM inference is increasingly dominated by KV-Cache storage I/O rather than computation." In disaggregated prefill/decode clusters, loading a turn's KV from external storage saturates prefill-side storage NICs while decode-side NICs idle; DualPath adds a storage-to-decode path (via RDMA over the compute network) and reports up to 1.87× offline and on average 1.96× online serving throughput over its in-house baseline. The significance for capacity planning: the paper treats KV as a placed, moved, and load-balanced byte stream — a storage-systems scheduling problem, not a model-serving one. It puts numbers on what the per-turn math above predicts: the prefix re-read is a mandatory data movement, and where you route those bytes decides throughput before FLOPs ever enter the picture.

TensorRT-LLM's KV Cache Manager V2³ is the same pattern in product form: a hot tier in HBM and a cold pool for offloaded KV, cold-page codecs for compressed cold pages, per-layer-group pool_ratio quotas, and cold-tier statistics moved into kvCacheIterationStatsByColdPoolGroup (the 1.2 release notes carry the pool-ratio and cold-tier-statistics breaking changes; the cold-page codecs and the V1-deprecation announcement landed in the v1.3.0 RC line, August 2026). V2 is the recommended architecture, and NVIDIA has announced V1 will be deprecated; hot/cold tiering is no longer exotic — it is the default-on control plane for exactly the re-read traffic this guide computes.

The mechanical gap: PCIe vs HBM

Whether the "cold tier" is host DRAM over the PCIe link⁴, a CXL expander, or a remote KV server, the bytes cross a fabric slower than HBM — usually PCIe. FAST NVMe-class: PCIe 5.0 x16 ≈ 64 GB/s raw, ~63 GB/s effective (32 GT/s × 16 lanes with 128b/130b encoding, per the PCI-SIG Gen5 spec). Set against HBM3e at 4.8 TB/s:

The ~75× gap is not a detail — it fixes the floor of every turn. Re-read timings, computed:

A page-in from the cold tier adds a tenth of a second to ~a second per turn — invisible in chat terms, catastrophic at agent cadence (a tool loop can turn around in under a second). This is why tiering placement policy (which pages stay hot) matters more than tiering capacity: the bytes that stay in HBM cost 76× less latency every time they're touched. NVLink-C2C-class links (2.25 TB/s+ per the NVLink figures) sit between the extremes and are the reason GB200-class integrated designs hold up better than PCIe-attached host offload.

The economics: tokens/joule with the I/O term

Chat decode economics: batch-1, one step reads weights + KV. Agent decode economics with the same 700 W card, computed per token:

The interesting line is not the J/token — it is that at 1M ~52% of an agent step's time is pure KV re-read (70.27 GB ÷ 3.35 TB/s = 21.0 ms of the 40.4 ms step; the other ~48% is the weights read — every microsecond of the step is memory traffic, none of it math). The GPU burns its full 700 W while doing 3.4% useful math. Energy per delivered agent token is dominated by an I/O term, not a compute term — the opposite of chat, where the weights read dominates and batching amortizes it. Once KV sits behind a slower tier, add the PCIe energy too: a full 1M-KV cold-tier restore costs ~1.12 s at PCIe 5.0 x16 — real joules for zero FLOPs.

When prefix caching kills the re-read

RadixAttention (SGLang)⁵, arXiv:2312.07104) makes the recompute half of the re-read nearly free: shared prefixes are stored in a radix tree, and the paper reports up to 6.4× throughput on prefix-heavy workloads — multi-turn agentic loops are its best-case pattern (turn n shares turns 1…n−1). Prefix caching kills prefill compute, and inside one GPU it keeps the hot KV resident so HBM re-reads flow at full speed.

When it can't: eviction under concurrency

Prefix caching cannot fix capacity. LRU eviction hits the moment the working set exceeds the hot tier — and the working set is exactly what this guide computes. On a 141 GB card (with weights on other cards of the node):

50 workers × 1M ctx = 3,513 GB of KV = ~25 × 141 GB cards or ~20 × 180 GB cards — just to hold the fleet's state, before a single FLOP of weights is placed. At 100k average the set is 351 GB — still 2.5 cards' worth of pure cache. Under a radix tree this is brutal: the moment worker 51 arrives, the LRU evicts precisely the long prefixes that made agentic turns cheap, and the next turn pays 0.11–1.1 s (per the PCIe table) restoring bytes the fleet just computed. Eviction pressure at agent cadence is a thrash loop, and concurrency is what triggers it.

What capacity planning actually looks like (the critical view)

"Agents run on the same infra as chat" is the hype line — usually followed by a tokens/s quote. The arithmetic says otherwise, and the honest sizing heuristic falls straight out of the numbers above:

1. Compute the per-agent working set: bytes/token (70.27 KB here, per the MLA figures) × context. Write it down; it is the single number that sizes everything.
2. Multiply by fleet size: that's your storage array's capacity. 50 × 1M = 3.5 TB of managed state — plan it like array capacity: hot HBM tier, cold DRAM/CXL/remote tier, eviction policy as the placement engine.
3. Provision movement, not FLOPs: fleet decode rate × working set = required read bandwidth (70.3 TB/s at 20 tok/s above). Check it against the aggregate HBM of the hot tier and the cold-tier fabric (63 GB/s per PCIe x16; ~50 GB/s per 400 Gb NIC) — this is DualPath's storage-NIC observation exactly.
4. Check the idle math before believing any tokens/s claim: if GPU utilization is 3.4%, the fleet is a disk array with expensive attached SIMD units.

The pattern is already assembling: DualPath routing the byte stream, TensorRT-LLM V2 hot/cold pools with codecs and quotas, HotInfra'26's memory-centric KV cache servers on CXL/PIM fabrics. The infrastructure implication — plan agent fleets like storage arrays, in working-set bytes — is not predictive speculation; it is the design assumption these systems were built on in 2026. Capacity failed quietly in the chat era because a conversation's state is trivially small. A concurrent agent fleet's state is terabytes, re-read continuously. Size it as such.

References