UNISON and the Paused Fleet: Why Agent KV Traffic Breaks Every Cache Policy You Know

Agent serving has a caching problem that neither LRU nor TTL can see, because it hides in the gap between two events. When an agent calls a tool, its GPU sits idle — but the session is not dead, and when the tool returns, the session resumes from its full KV prefix. A September 2026 paper from Fudan University, UNISON (arXiv:2609.09643, He, Li, and Zeng, submitted Sept 9, 2026)¹, builds a near-memory scheduler around exactly this blind spot, and it is worth reading less for its headline numbers than for how precisely it names the workload failure: a session parked on a tool wait is treated as "cold" by every recency or timeout policy, and is evicted seconds before the tool returns. This guide does the working-set math, verifies the paper's mechanism against the arXiv text itself, prices the silicon, and is brutal about which gains a radix-tree could have given you anyway.

The paused-fleet working set: 703 GB that must sit somewhere

The core arithmetic first, with the site-verified KV-cache numbers. DeepSeek-V3 MLA stores 576 latent elements per token per layer across 61 layers in bf16 — 70,272 B ≈ 68.6 KiB per token². A single agent at 100k context therefore carries:

Now the paused fleet. One hundred agents, each parked at 100k context while their tools run:

Computed in Python: 100 × 100,000 × 70,272 B = 702.72 GB (654.5 GiB). That is roughly nine 80 GB-class GPUs of pure KV state, occupied by sessions that are generating nothing, paying no attention FLOPs, and yet must resume with byte-exact prefixes seconds to minutes later. This is the storage-array view of agentic inference we developed in the KV-tiering guide: the fleet's dominant resource is not compute but resident bytes, and tool-wait duty cycles decide where those bytes live.

Here is why both standard cache-policy classes fail on exactly this state:

• Recency-class (LRU) evicts the wrong thing. LRU ranks by last access. A tool-waiting agent has not touched its KV since its last completed turn — possibly minutes ago. In a contended pool, it looks older than a chatty user who pinged the endpoint two seconds ago. LRU kills the paused agent first.
• Timeout-class (TTL) evicts it on schedule. A TTL policy expires sessions after a fixed idle window. But tool latency is heavy-tailed: a code-execution sandbox or a browser step can take seconds or tens of seconds. Any TTL short enough to be useful also expires the sessions whose tools happen to be slow — which UNISON's own framing calls treating "a live wait as a cold, discardable unit."¹
• The failure is structural, not tunable. There is no threshold on recency or idle time that separates "waiting on a tool, will return" from "session over" using access history alone, because the distinguishing information — the loop's mechanism, the fact that a tool call implies a return — lives in the job structure, not in the access log.

A concrete failure mode: picture a coding-agent fleet whose sessions run 30–60 turns with tool gaps of 2–30 s, sharing a pool sized for half of them. Under any burst, LRU evicts exactly the sessions mid-tool-call — the ones with the most banked context and the highest re-prefill cost if lost. The TTL instinct (pin everything for 30 s) inverts the failure but solves nothing: it strands capacity on sessions that already ended and still loses the agent whose tool took 45 s. Both policies fail because they ask the access log a question only the job structure can answer.

The paper's own related-work section makes the same point at the system level: PagedAttention pages the pool like virtual memory and SGLang reuses prefixes through a radix tree, but "both evict by block recency, so a session alive on a tool wait" is structurally misranked.³

Two refinements matter before the silicon talk. First, where the 703 GB sits is a tiering decision, not an afterthought: the options are (a) keep it in HBM/VRAM — impossible at fleet scale, that memory is needed by the sessions actually decoding; (b) drop it and re-prefill on resume — costing ~7 GB of read-and-recompute per resume, the worst case; or (c) host-memory spill and hierarchical stores, which is where the paper aims its idle-window migration. Second, the policy question — who stays in the fast tier, who spills, who is dropped outright — is exactly the decision recency and timeout proxy badly. Put the tier decision in absolute numbers: a CXL-attached DDR5 pool currently delivers on the order of a few GB/s per device of effective spill bandwidth against HBM's multiple TB/s — three orders of magnitude apart. That is why when you move a 7 GB session matters as much as whether: a migration done inside a 10 s tool gap at idle DDR bandwidth is free; the same migration started just as the tool returns competes with the resume prefill and is a pure loss. Idle-window scheduling is not an optimization on tiering — it is the difference between tiering being viable and tiering being a latency bug.

Each of the three tiers above has a cost the policy controls: misrank a live session to tier (b) and you pay a full re-prefill on its next turn; misrank a dead session to tier (a) and you strand fast-tier capacity until something notices. UNISON's claim is that making these decisions event-driven, with loop-mechanism information (tool call issued, tool gap length, turn index), strictly dominates both proxy classes — and that the decision belongs in silicon near the memory hierarchy because that is the only place it can be made without paying data-path cycles for it.

UNISON's headline numbers at a glance

• 1,415 sessions / 33,596 turns, three model families, coding and general-mission traces — modest by production standards, sizable for a scheduling paper¹
• Hit rate +0.3% to +23.1% vs. the best non-oracle alternatives — the width of that range is the trace-contention story in one number¹
• AMAT −22% to −51% — average memory access time, the metric that prices tier placement¹
• TTFT −58% to −89% on long-horizon traces — a hit avoids re-prefilling a ~7 GB 100k prefix outright¹²
• 0.169 mm² / 13.6 mW / 150 MHz at 28 nm, 64 sessions per core, Kendall τ > 0.998 vs. floating-point reference¹³

Every one of those comes from the arXiv abstract or HTML full text, fetched and checked for this piece — footnote at the bottom of each claim. Now the mechanism.

UNISON verified: SPEAR + TIDE, one shared ranking

Reading the actual paper (abstract and HTML full text)¹³, the acronyms check out and the architecture is genuinely joint, not two bolted-together heuristics:

• UNISON = Unified Native Inter-turn Session Orchestration Nexus — an event-driven near-memory scheduler for session KV residency.
• SPEAR = Survival-Penalty Eviction for Agent Return-gap. It decides who leaves the pool, from a gap average plus a turn-indexed hazard: sessions with historically long tool returns and many turns already banked survive longer, because survival curves say they will come back.
• TIDE = Tiering in Idle-window DMA Events. It decides who sits in the fast tier, spending the observed wait time itself as a DMA budget — while the agent is paused, its blocks can migrate between hierarchy tiers using bandwidth that is otherwise idle.

Number-checking the claimed losses is also sobering: even with the joint policy, the best-case hit-rate gain on some traces is a third of a percent — the ceiling on any scheduler's value is set by how often the pool is actually contended, and agent fleets with low concurrency simply do not contend. The TTFT gains, by contrast, are enormous precisely because the counterfactual is re-prefill: on a long-horizon trace, one saved prefix is an entire 100k-token prefill avoided, and there is nothing incremental about that saving.

• The key co-design claim: SPEAR and TIDE share one live ranking, and the paper includes a structural-necessity analysis arguing the unified near-memory design "cannot be decomposed into independent IPs or realized in software without re-introducing documented failure modes."¹ Note that last clause is the authors' claim — more on that below.

The evaluation, verified against the abstract: coding and general-mission benchmarks across three model families, totaling 1,415 sessions and 33,596 turns. The joint policy is reported as the best non-oracle entry on every trace, with hit rate up 0.3% to 23.1%, AMAT (average memory access time) down 22% to 51%, and TTFT down 58% to 89% on long-horizon traces.¹ The breadth of those hit-rate ranges is itself informative: at the low end (+0.3%), a good policy saves almost nothing — the trace simply did not have contention; at the top end, the baseline was losing a quarter of the pool to misranked live sessions. There is also a vLLM-based production-stack validation section comparing against LRU across tool-call-frequency schedules.³

The survival framing deserves emphasis because it is the paper's real conceptual move. Access-history policies are point estimates: "how recently was this touched?" SPEAR instead fits a survival function over the return gap — how much longer, given this session's history and turn index, is its tool likely to take? — and penalizes eviction accordingly. That is the same statistical shift that took CDN caching from LRU to learned, relaxed-Belady-style policies (the paper cites that lineage in its related work ³). The turn-indexed hazard encodes an empirical regularity of agent loops: a session that has returned from 40 tool calls will almost certainly return from its 41st, which is precisely the signal an access log cannot express and a timeout cannot represent.

Interpretation: the AMAT and TTFT numbers are the load-bearing ones for serving economics. A hit that avoids re-prefilling a 100k-token prefix does not save the agent's next GPU second — it saves the entire re-prefill of the sequence, which at 7 GB per session is the difference between "resume in one DMA pass" and "re-read and re-compute a book-length context."

One more sanity check on the trace claim: "best non-oracle entry on every trace" is doing real work in that abstract sentence. It means UNISON beat every proxy-class baseline everywhere — but an oracle with perfect future knowledge still wins, and the paper does not claim otherwise. That is the honest framing: the scheduler closes most of the distance from LRU toward Belady-class foresight, and the oracle gap tells you how much mechanism information is still missing (chiefly: some tool outcomes genuinely are unpredictable, and no hazard model recovers that).

Micro-silicon economics: 0.169 mm², 13.6 mW, 150 MHz

The scheduler core is a 28 nm CMOS block: 0.169 mm², 13.6 mW, 150 MHz, covering 64 sessions, and it reproduces the floating-point ranking at Kendall τ > 0.998 against the reference.¹³ What class of silicon is that? Tiny. For scale, that is smaller than a single HBM4 PHY and drawing less power than a dimmable LED — embedded-controller class, the kind of block that ships by the dozen next to a memory controller. Yet it arbitrates residency for a hierarchy that holds, in our paused-fleet frame, hundreds of gigabytes.

The seeming paradox — mW-scale logic moving terabyte-scale traffic — dissolves once you see what a scheduler actually moves: pointers and decisions, not bytes. The DMA engine and the hierarchy move the bytes; SPEAR + TIDE only decide which blocks those engines touch and when. A residency decision is a few hundred bits of state per block; the block it governs is 70 kB or more. The control/data asymmetry is five to six orders of magnitude, which is the entire economic argument for near-memory scheduling: put pennies of logic next to memory so that dollars of bandwidth are never spent on the wrong blocks. There is a useful sanity check on that asymmetry: the paper's own numbers: at 64 sessions and 150 MHz the mean scan latency is 2.00 µs, worst case 3.14 µs — just 0.31% of the smallest observed tool gap (1 ms) and far below the 3.7–7.2 s medians, so the ranking is effectively instantaneous relative to the events it ranks. A host-side scheduler would burn CPU interrupts and PCIe round-trips to reach the same decision with strictly worse placement to act on it.

The paper's own hardware framing is "a negligible overhead relative to the KV hierarchy it manages"¹ — and for once the marketing-adjacent phrase survives contact with the spec sheet: 13.6 mW is roughly one-hundred-thousandth of a single Rubin-class GPU's envelope (~1,400 W). Even multiplied per rack to cover thousands of sessions (at 64 sessions per core, ~32 cores for 2,000 sessions ≈ 0.44 W total), the policy logic costs less than a case fan. The bytes it shepherds cost megawatts. That ratio — not any single hit-rate number — is the argument the paper actually lands, and it is the ratio any future "KV management in silicon" proposal should be judged by.

The Kendall-τ result is the credibility check — the fixed-point, 150 MHz version of the ranking agrees with the floating-point reference near-perfectly (Kendall τ > 0.998), so the silicon is not an approximation ghetto; it is the policy.

The software-only counterfactual: what RadixAttention already gets you

Be brutal here, because a chunk of UNISON's headline territory is reachable in software today. RadixAttention (arXiv:2312.07104) — the SGLang prefix-cache — organizes KV as a radix tree over token sequences and reuses shared prefixes across requests, yielding up to 6.4× higher throughput on prefix-heavy multi-turn tasks — the paper attributes this to the full SGLang system (radix reuse + compressed FSMs + runtime), not the tree alone.⁴ An agent session is precisely a prefix-heavy workload: each turn inherits nearly the whole prior context, so a kept prefix is a full-length re-prefill avoided. The paper itself acknowledges this lineage (its related work cites the radix runtime), and its own workload model states the resident KV is "prefix-heavy because later turns inherit most of their context from earlier ones, so a hit avoids re-prefilling the full prefix."³

That distinction is not rhetorical: it determines what you deploy. A serving team can capture the ranking half of UNISON (survival-penalty eviction) as a patch to its runtime's block manager — no new hardware, no schedule risk. The data-path half (idle-window DMA tiering) requires either CXL host-spill infrastructure or the paper's near-memory block, which is exactly what a production adoption would have to wait for.

So split UNISON's gains by mechanism:

• Rank-new-blocks-first gains — software-replicable. Much of SPEAR is policy: knowing that a tool-waiting session will return, from gap statistics and turn index. Nothing about that ranking requires 28 nm silicon. A serving runtime could implement survival-penalty eviction in its host-side scheduler tomorrow — SGLang already has the radix-tree block granularity to hang it on. If UNISON's hit-rate gains come mostly from better ordering of evictions (and the LRU baselines in the vLLM section suggest exactly this), the software counterfactual captures them.
• Data-movement offload gains — these are the real hardware story. TIDE's contribution is not the ranking but the idle-window DMA: while the fleet is tool-waiting, the memory system's read bandwidth is idle, and migrating blocks between tiers then is free in a way a host-side copy (which burns PCIe/CPU cycles on the serving node) is not. The paper's structural-necessity claim is aimed at this path: software tiering re-introduces the failure modes because the host is the one paying the migration tax with data-path cycles.¹ The AMAT reductions are where this should show up — and it is the part you cannot get from a smarter eviction order alone.
• The honest split we can verify: the paper reports its software ablations as re-introducing "documented failure modes" rather than as a clean X%-of-gains-is-ranking breakdown.¹ That is a weaker form of evidence than a decomposition table would be, and a skeptical reader should treat the "impossible in software" framing as unproven until someone ships a survival-penalty eviction policy in vLLM and measures the delta.

When not to want this: single-user setups, short-context chat models, or fleets where tool calls are masked synchronous API hops (sub-100 ms) — the working set never separates from compute, and an idle-window DMA engine would be scheduling nothing. The pattern earns its silicon only when pause-resume duty cycles and prefix sizes make residency the binding constraint.

Verdict

UNISON is the most credible piece yet of a specific pattern: moving agent-session KV policy out of the serving runtime and into cheap silicon next to the memory it manages. What it gets right, verified: the workload diagnosis (recency and timeout proxies provably misrank live tool waits), a power/area budget you can actually check (13.6 mW is a rounding error against one HBM stack's idle draw), and a mechanism — idle-window DMA tiering — that is genuinely data-path-native rather than a scheduling heuristic with hardware color.

The caveats, equally on the record. It is a research prototype with no tapeout; the trace class is academic agent benchmarks (1,415 sessions), not a production fleet with adversarial tenants; the "+0.3% to 23.1%" range shows gains are extremely trace-dependent — where there is no memory contention, no policy can win; and the software-impossibility claim is the authors' framing, tested against LRU rather than against a strong host-side survival-penalty baseline. The RadixAttention counterfactual remains the null hypothesis any production evaluation must beat: if your eviction is LRU, prefix reuse plus survival penalties in software may already be most of the win.

When does this pattern matter in production? Multi-tenant agent fleets at high pause-resume duty cycles — idle-window DMA only pays when (a) tool-wait gaps are frequent and long enough to be a usable migration budget, (b) the working set of paused sessions rivals the fast-tier capacity (our 703 GB fleet — nine GPUs of sleeping state — fits this), and (c) hits save full re-prefills rather than partial ones. If your agents are chatty with sub-second tool calls, UNISON's trace-class says so itself: +0.3%. What would change our mind, concretely: (1) a production-trace replay — say, ten thousand real agent sessions from a coding-assistant fleet, with per-tenant contention — reproducing the AMAT deltas; (2) a host-side survival-penalty baseline in vLLM or SGLang published alongside, so the software-only null hypothesis is measured rather than asserted away; and (3) one of the memory vendors confirming the block integrates next to a CXL tier or an HBM controller without a custom ASIC program. Until those exist, the honest reading is: right diagnosis, cheap and checkable silicon story, unproven necessity claim.

Until a hyperscaler publishes the production ablation, treat UNISON as the best worksheet for the paused-fleet math — and treat the silicon endpoint as promising, unproven, and cheaper than you probably assumed.