Your Agent Fleet Is a Storage Array: The KV-Cache Tiering Math Behind Agentic Inference

Long-context agent loops can put substantial pressure on KV-cache capacity and transfer bandwidth. The calculations below are illustrative lower bounds for uncompressed BF16 KV under full attention. Actual serving depends on attention kernels, batching, cache residency, quantization, and model parallelism. The model configuration and sources are linked in the KV-cache guide.

The workload shape: long-prefix re-reads plus small increments

An agentic loop may call a tool and continue from a long prefix. Consider a 100k-token context, a 2,048-token tool result, and a 512-token reply:

• Decode: with full attention, each new token attends to the prior prefix. The approximately 100k-token KV is read at each of 512 decode steps, subject to implementation details and growing context.
• Increment: only 2,048 + 512 tokens of new KV get computed and written.

Reference geometry: DeepSeek-V3 MLA¹, with 512 latent plus 64 RoPE elements per token per layer, 61 layers, BF16 → 70,272 B ≈ 68.6 KiB per token, assuming this cache layout. Its model card advertises 128K context; the 1M-token fleet scenario below is an explicit extrapolation of this geometry, not a supported DeepSeek-V3 configuration.²

That is the cache capacity for one session at the start of the turn, before allocator and block overhead.

Per-turn HBM bytes: agent vs chat

If this cache remains in HBM and the kernel reads the full prefix on every decode step, approximate read volumes are:

This example yields 25× as many KV-read bytes for the agent turn. It does not establish a 25× token cost or latency: batching, kernel efficiency, memory placement and the separate cost of model weights all matter.

Incremental prefill of the tool result also computes new KV. Its execution time cannot be inferred reliably from peak FLOP/s alone, especially for a distributed MoE model.

A hypothetical 50-worker, 1M-context fleet

For capacity planning, extrapolate the same BF16 MLA layout to 50 simultaneous workers at 1M tokens each. Again, this exceeds DeepSeek-V3's advertised context length and is a hypothetical cache-sizing example.

If all KV is HBM resident and full attention reads it at each step, these per-worker decode rates imply the following aggregate HBM KV-read demand:

A 141-GB H200 has a published 4.8-TB/s HBM peak.³ Dividing 3.514 TB by 141 GB gives about 25 H200 cards as a strict KV-only capacity minimum, with no space for weights, fragmentation, or runtime state. At 20 tokens/s per worker, 70.3 TB/s divided by 25 is 2.8 TB/s per card, about 59% of the theoretical H200 HBM peak. That is a bandwidth budget, not a measured utilization or throughput prediction.

HotInfra'26⁴ proposes memory-centric KV-cache servers on a spectrum that includes CXL expanders and near-memory processing. That design proposal supports the importance of KV placement; it does not measure 3.4% GPU utilization for the scenario here. See also the scale-up networking analysis.

The tiering pattern

DualPath⁵ targets a turn-boundary KV load from external storage in disaggregated prefill/decode systems. It adds a storage-to-decode path and RDMA transfer to prefill engines, reporting up to 1.87× offline throughput and 1.96× average online throughput against its in-house baseline. These are results for that system and workload, not a general speedup for every agent fleet. Distinguish this one-time turn-boundary load from the repeated HBM reads during decode calculated above.

TensorRT-LLM's KV Cache Manager V2⁶ supports hot HBM pools and cold-page handling. Its use_kv_cache_manager_v2 default is auto: documented model-specific choices use V2, while models without a preference fall back to V1. V2 is therefore not universally enabled by default. Its cold-pool functions concern cache placement and transfer, not a way to read cold storage at HBM speed for every attention step.

The mechanical gap: PCIe vs HBM

For one possible offload path, PCIe 5.0 x16 provides roughly 63 GB/s per direction after 128b/130b encoding, before packet and software overhead.⁷ H200 HBM's published peak is 4.8 TB/s. Comparing these peak figures:

The approximately 76× ratio compares interface peaks, not measured end-to-end transfer rates. Ideal transfer lower bounds are:

Those PCIe times apply only when the full cache needs to be restored; a cache hit has no such transfer, and transfers may overlap with compute. NVIDIA specifies 900 GB/s bidirectional NVLink-C2C for Grace Blackwell and up to 1.8 TB/s for Vera Rubin, rather than 2.25 TB/s.⁸ The link rate alone does not determine application-visible KV reload latency. See the NVLink figures.

Energy and cost: what the bandwidth estimate cannot establish

The bytes above are useful for capacity and bandwidth budgeting. They do not determine GPU utilization, joules per token, or price per token. That requires a specified model placement, batch size, measured power, kernel throughput, and actual cache-hit rate. A prior single-card table mixed a 65-GB weight read with DeepSeek-V3's 37B active parameters and ignored its distributed 671B total weights; it cannot describe a real DeepSeek-V3 deployment.²

What prefix caching saves

RadixAttention in SGLang⁹ reuses KV blocks for shared prefixes. When a prefix remains resident, the next turn avoids recomputing its KV in prefill. Decode attention still reads the resident prefix KV; caching does not eliminate those HBM reads.

When it can't: eviction under concurrency

Prefix caching remains subject to memory capacity. For an idealized 141-GB card dedicated entirely to KV:

50 workers × 1M tokens require 3.514 TB of KV, equivalent to roughly 25 × 141-GB or 20 × 180-GB cards as capacity lower bounds. Real systems need more for weights, fragmentation, and runtime buffers. Whether another worker triggers eviction depends on placement and policy; it cannot be inferred from a fleet-wide count alone.

What capacity planning actually looks like (the critical view)

"Agents run on the same infra as chat" is the hype line — usually followed by a tokens/s quote. The arithmetic says otherwise, and the honest sizing heuristic falls straight out of the numbers above:

1. Compute the per-agent working set: bytes/token (70.27 KB here, per the MLA figures) × context. Write it down; it is the single number that sizes everything.
2. Multiply by fleet size: that's your storage array's capacity. 50 × 1M = 3.5 TB of managed state — plan it like array capacity: hot HBM tier, cold DRAM/CXL/remote tier, eviction policy as the placement engine.
3. Budget two distinct paths: resident KV is read from HBM during decode; missed or evicted prefixes are loaded over the storage/host path at turn boundaries. Estimate miss rate and transferred bytes separately.
4. Benchmark the deployment: check throughput, utilization, and energy with the actual model, context distribution, cache policy and hardware.

DualPath, TensorRT-LLM V2, and the HotInfra proposal each address parts of this placement problem. The exact break-even point for offloading or adding GPU memory is workload-specific; the equations here provide a starting budget, not a universal serving recommendation.

References