BOOST and the End of Prefetch: Why Grace Hopper Wants Both Memory Tiers at Once

A September 2026 paper from Georgia Tech, NVIDIA Research, and Stanford — BOOST (arXiv:2609.13592, Saxena, Ju, Taneja, Tsai, Jaleel, Kozyrakis, Qureshi)¹ — makes a claim that sounds like routine serving-systems tuning and is actually a small paradigm kill: host-memory KV/weight tiering should not be built on prefetching, because prefetching mechanically reduces the usable bandwidth of the fast tier it is trying to feed. The runtime serves both tiers concurrently, in proportion to their bandwidth, and measures +31% average throughput in vLLM on Grace Hopper while prefetching measurably hurts latency — TPOT degrades 6% at iso-batch¹²³.

The hype-adjacent framing you will see elsewhere is "31% free throughput from RAM you already own." The hype-resistant version is narrower and more interesting: on this specific class of tightly-coupled CPU-GPU systems, the host tier is a ~10% bandwidth peer and a ~10% capacity annex, and the entire design space of tiering systems built on "move data to the fast tier before use" pays a write tax it can never recover. This guide recomputes the ledger in Python (you can run the same arithmetic), separates the throughput story from the latency story, and draws the silicon boundary the claim cannot honestly cross — which matters because it is the same boundary our UNISON analysis hit from the other side: UNISON wanted dedicated scheduler silicon; BOOST gets real measured wins from 800 lines of Python in a serving engine, on real hardware. The tension between those two claims is the story.

1. The claim, and what was measured on what

Provenance first: Georgia Tech + NVIDIA Research + Stanford, integrated into vLLM v0.17.0 with roughly 800 lines of Python/PyTorch, no kernel edits — it works through mmap/mbind page placement and a modified KV-pool manager; FlashAttention-3 and closed-source cuBLAS kernels run untouched². Evaluation hardware is a Grace Hopper GH200: 96 GB HBM, 480 GB CPU memory, spec peak HBM bandwidth 3.63 TiB/s and measured in-practice 3,330 GiB/s, measured C2G (chip-to-GPU, over cache-coherent NVLink-C2C) read bandwidth 350 GiB/s — that measured pair gives the bandwidth ratio α ≈ 10% that the whole paper turns on². We verified every number below in the arXiv HTML full text; the trade-press pickup (Semiconductor Engineering) quotes the abstract accurately³.

Three headline numbers, each of which decomposes differently, so it is worth keeping them apart:

• +31% average throughput, up to 40% at max-batch scaling vs. HBM-only serving (measured)².
• +4.3% TPOT at iso-batch — same batch, fewer milliseconds per token, purely from added bandwidth².
• −6% TPOT for prefetching under the same iso-batch conditions — the incumbent paradigm loses latency on the workload it is supposed to help¹.

Note the asymmetry immediately: 31% is mostly a capacity effect, 4.3% is the bandwidth effect, and −6% is the cost of the prefetch paradigm itself. The rest of this article is the arithmetic that makes those three numbers inevitable rather than impressive.

2. The bandwidth ledger: why prefetch can never use host bandwidth during decode

The paper's core observation is an accounting identity, and accounting identities are the best antidote to hype there is. A prefetching tier writes every host-resident byte into HBM before the SMs read it. That write competes for the HBM interface with the demand reads of the decode step. The host link's bandwidth is spent pulling data across C2C, and then a second time as write traffic on the HBM pins:

• Read 1 GB from host over C2C: consumes C2C bandwidth the system otherwise never uses during decode.
• Write that 1 GB into an HBM staging buffer: consumes HBM write bandwidth.
• Read it back as a demand load: consumes HBM read bandwidth.

So per delivered byte, a prefetcher pays HBM write + HBM read + C2C, and the C2C spend is the only new resource — everything else is subtracted from the tier you were trying to offload. Worse, the writes are recurrent: the offloaded working set is far larger than any staging buffer, so staged bytes are evicted after use and re-fetched at the next decode iteration². The paper measures the damage at 211 GiB/s of demand-read bandwidth lost on Llama-3.3 70B in vLLM² — about 6.3% of the 3,330 GiB/s baseline:

# Grace Hopper, paper-measured (arXiv:2609.13592v1, Sec. 5.1)
HBM  = 3330   # GiB/s measured demand bandwidth
C2G  = 350    # GiB/s measured chip-to-GPU read bandwidth
alpha = C2G / HBM            # 0.105 -> the paper rounds to 10%
 
prefetch_loss = 211          # GiB/s demand bandwidth lost to prefetch (Fig. 3)
print(alpha, prefetch_loss / HBM)   # 0.105, 0.0633

Now the staging-buffer ledger, which is where the "5+ GB aggressive buffer" folklore meets reality. Take Llama-3.3 70B in FP8 — call it 70 GB of weights swept by every decode step (KV adds more on top). A typical tiering policy offloads one of every K layers to host and prefetches asynchronously. Run the write arithmetic per decode step:

W = 70.0   # GB of FP8 weights read per decode step (KV on top)
for K in (10, 8, 5):
    staged = W / K          # GB written into HBM staging per step
    print(f"K={K}: {staged:.1f} GB written/step = "
          f"{100*staged/W:.1f}% extra HBM write traffic vs weight reads")
# K=10: 7.0 GB written/step = 10.0% extra HBM write traffic
# K=8:  8.8 GB written/step = 12.5% ...
# K=5: 14.0 GB written/step = 20.0% ...

A 5–6 GB staging buffer cannot hold even the K=10 offload slice (7 GB), so each decode iteration stages, evicts, and stages again — recurring HBM write traffic in the same ballpark as the α it was supposed to add. That is the structural reason an ideal, perfectly overlapped prefetcher still cannot win: with fraction α of data staged, effective demand bandwidth is bounded at HBM × (1 − α), because every staged byte must land in HBM. Concurrent dual-tier reading attains HBM × (1 + α). The ratio of the two bounds is (1 + α)/(1 − α)²:

for a in (0.03, 0.105, 0.44):
    print(f"alpha={a:.3f}: CAP/prefetch bound = {(1+a)/(1-a):.2f}x")
# alpha=0.030:  1.06x  (GB200-class: 2 GPUs share one CPU)
# alpha=0.105:  1.23x  (measured GH200; paper states 22% at alpha=10%)
# alpha=0.440:  2.57x  (projected Vera-class, 900 GB/s C2G)

This is the cleanest way to state the paradigm result: prefetching converts the second tier from a bandwidth source into a bandwidth tax. It is not a tuning failure, an overlapped-DMA failure, or a cleverness failure — it is what the ledger says must happen when every host byte must first become an HBM byte.

3. Why concurrent proportional access wins: add the tiers, don't average them

BOOST's alternative is what the paper calls CAP access — concurrent and proportional determine: within every wave of threadblocks, both tiers are read simultaneously, with exactly α/(1+α) of accesses to host and 1/(1+α) to HBM. The proportionality condition is old (bandwidth-proportional page placement dates to 2015); the concurrency within a wave condition is the new one, and it exists because modern GPUs use 2 MB pages. A wave touches tens of 2 MB pages, not the thousands of 4 KB pages that made 2015-era random proportional placement self-balancing within a wave. Random placement that is proportional on average swings wildly per wave: some waves oversend the host tier and stall on it while HBM idles; some waves touch host barely at all. The paper's controlled sweep is blunt: speedup peaks at 7.4% at a host share of 8.3% (near the proportional 9.1%), and a host share meaningfully above proportional is a 6.7% slowdown². Proportion with no concurrency = no gain; concurrency with skewed proportion = loss. Both properties are load-bearing.

Two mechanisms deliver it without kernel edits. For static weights, Modulo-based Page Placement (MPP): allocate the first K pages to HBM, the (K+1)-th to host, repeat — the access ratio is deterministic in every wave with zero variance, and CTAs stay un-split across tiers (a CTA with mixed-tier footprints stalls its SM on host loads). For dynamic KV, the Data Shape Remapper (DSR) maps whole KV heads to tiers — one head of an H-head model to host gives head-granular control, with a tensor-layout reorder so heads stay contiguous and page-aligned. The runtime reserves about 9.6 GB (2% of the 480 GB host side) in 1 GB pages and performs no migration at all — data is read where it lives, cacheline-granular, over cache-coherent load-store semantics².

The bandwidth bound is then simply additive. Both tiers saturate per wave:

print(f"dual-tier bound = HBM + C2G = {3330+350} GiB/s vs {3330} HBM-only "
      f"-> {(3680/3330-1)*100:.1f}% max")
# 3680 GiB/s -> 10.5% max bandwidth gain at iso-batch (paper's ~1.1x framing, alpha 9-11% per SKU)

That is the honest ceiling: ~11%, not 31%. So where does 31% come from? Keep the two effects separate — that separation is the article.

4. The 4.3 / −6 / +31 split: latency buys 11%, capacity buys the rest

The iso-batch TPOT experiment fixes the batch (10 requests for Llama-3.3 70B, 88 for Qwen3-Next 80B) so only bandwidth can change the result. There BOOST delivers 4.3% average TPOT improvement — the paper's geometric mean, which is the right aggregator when the per-model numbers are 4.5% (MoE) and 4.2% (dense)². Prefetching under identical conditions delivers −6% — meaning BOOST ends 11% ahead of the incumbent, and the MoE case is worse still for prefetch: dynamic expert activation times desynchronize the prefetch pacing, misses its timing window, and costs the MoE 9.3% TPOT². The measured bandwidth check: BOOST's total (HBM+C2G) utilization improves 3% over HBM-only and 12% over prefetching².

Now throughput. In high-throughput mode the serving system auto-scales batch size to fill available memory — and this is the capacity lever the abstract's framing sentence makes explicit: if 20% of HBM space is KV, then 10% extra capacity from the host tier grows the KV space by 50%, allowing up to 1.5× the concurrent requests¹. Sanity-check that decomposition numerically:

total = 1.31          # measured average throughput gain
concurrency_share = 1.04   # paper: concurrent access adds ~4%, no capacity-only mechanism does
print(f"implied capacity factor = {total/concurrency_share:.2f}x")  # 1.26x

So the honest read of "+31%": roughly a 1.26× capacity-driven batch-size effect (more resident KV and weights → more concurrent requests), compounded by a 1.04× per-request bandwidth effect. And the batch-size headroom also absorbs host slowness: at high batch, per-request decode is more latency-tolerant (some requests wait anyway), so paying a slower average access cost matters less than paying it at exactly-fixed batch. That is mechanically why prefetching still gains +17% throughput (capacity is capacity, even through a lossy paradigm) but loses TPOT — and why BOOST's edge over prefetching is 15% in throughput but 11% in TPOT¹². Whoever quotes "31% faster LLMs" has collapsed an iso-latency measurement into a batch-scaling measurement; they are different axes.

Two anti-hype details from the sensitivity sections deserve equal billing with the headline:

• Batch size is a real boundary condition. The dense model's kernel-level speedup decays from 7% at batch 8 to 6% at batch 64 and turns into a 0.5% slowdown at batch 256 — at large batch, multiple CTAs share a weight tile and L2 reuse matters, and Grace Hopper does not L2-cache GPU-to-host loads, so host reads bypass the reuse path². MoE shows the inverse: −2% at batch 16 (too few threadblocks to balance the ratio), +6% at batch 256. BOOST is a serving regime tool, not a universal constant.
• Weighted interleave — the "obvious" alternative — already gets you most of the throughput and none of the latency. Weighted-interleave placement raises throughput 30% on capacity but scores 0.99 on iso-batch TPOT: proportional-on-average without per-wave concurrency². If your workload only cares about tokens-per-second at saturated batches, you may already own most of BOOST's throughput win.

5. The silicon scope guard: NVLink-C2C is the claim, PCIe is not

This is where most of the secondary coverage will be wrong, so we state it as a hard rule. α ≈ 10% because Grace Hopper's host LPDDR5X sits behind a cache-coherent NVLink-C2C port measured at 419 GiB/s peak / 350 GiB/s sustained against HBM's 3,330 GiB/s measured (3.63 TiB/s spec)². Per SKU the paper quotes C2G read at 450 GB/s against HBM at 4,000–4,900 GB/s, hence "9% to 11%"¹. Roughly speaking, host read bandwidth is a tenth of HBM — only if the host is that close. The distinctive Grace Hopper property is that the host tier is a near-peer of the interconnect: ~512 GB/s-class LPDDR5X through a 600-GB/s-class coherent C2C port, i.e. nearly all the host bandwidth is deliverable to the GPU.

A PCIe-attached x86 host does not have this property at all. PCIe Gen5 x16 peaks at 64 GB/s (raw, ~50 GB/s realistic) against 4–8 TB/s of HBM: α ≈ 1–2%, an order of magnitude thinner. The bound arithmetic still holds directionally — prefetchers lose (1−α)/(1+α) there too — but the prize shrinks to low single digits, while the fixed costs (staging buffers, migration control, pinning) stay. Worse, PCIe lacks cache-coherent load-store semantics, so the "read it where it lives, ordinary loads" trick does not exist; you are in DMA-and-wait territory, which is why the prefetch paradigm exists on PCIe systems in the first place. The paper itself draws this line: "unlike traditional PCIe-connected systems where one CPU drives multiple GPUs, we focus on tightly-coupled CPU-GPU systems like GH200"².

Extend the guard across product lines rather than letting the marketing extend it for you:

• GB200: two Blackwell GPUs per Grace CPU — the CPU bandwidth is now shared, and the paper puts α around 3%². The dual-tier prize nearly vanishes even though it is still NVLink-coherent.
• Vera-class future systems: projected 900 GB/s C2G, α up to 44% — where the paper projects BOOST's advantage over prefetching growing to 60% and its own gains approaching 3ײ. Projection, not silicon; treat accordingly.
• PCIe x86 + DDR5: worst case for dual-tier reading. Prefetch-style tiering there is not a mistake — it is the only option the interconnect leaves.

If a vendor deck shows a Grace Hopper bar chart next to a PCIe server bar chart with the same "+31%" caption, that deck is lying twice.

6. The UNISON contrast: same workload diagnosis, opposite cost model

Our UNISON piece covered the near-memory scheduler for agent KV on tool-wait sessions: the paused-fleet working set (100 agents at 100k context ≈ 703 GB of pure KV state) that recency and timeout policies misrank, addressed by dedicated 28 nm scheduling silicon (0.169 mm², 13.6 mW at 150 MHz, 2.00 µs mean per 64-session scan). The honest caveat we flagged there, and now flagged harder: UNISON's evaluation runs on a homegrown trace-driven model of a Rubin-class GPU — not real silicon⁴. BOOST's evaluation runs on a real GH200 running real vLLM, real FlashAttention-3, real cuBLAS².

That contrast is a genuine open question in the tiering stack, and neither paper answers it alone. UNISON's claim is that tier-placement scheduling is event-driven, latency-critical, and belongs in silicon near the memory controller. BOOST's claim is that tier-bandwidth extraction is a placement problem a 800-LoC runtime solves on commodity hardware today, with no kernel changes and no new silicon. These are not actually the same claim — UNISON schedules when/where KV lives across a tier hierarchy under agent duty cycles; BOOST decides which tier each byte is read from so both saturate simultaneously — but they do compete for the same operator budget (engineering attention, adoption risk), and the burden of proof currently sits with the silicon camp, whose splashiest 2026 result is still simulation-based. If your decisive argument for near-memory silicon is "the scheduling decisions are too fine-grained for host software," you now have to beat a system that hit a 7.3% hand-kernel bound with 0 lines of kernel code, 96% of the way, from Python².

The counter-argument survives too, and it is worth keeping BOTH alive rather than declaring a winner: BOOST cannot help a paused fleet — its mechanisms assume actively-decoding kernels with steady per-CTA access streams; it has nothing to say about tool-wait residency, and its wave-awareness dissolves when the waves stop. The cheapest new HBM is the host DRAM you already own — our HBM4-shortage economics argument — but whether the host DRAM serves as a bandwidth peer (BOOST's regime) or as cold storage for parked sessions (UNISON's regime) depends on the workload's duty cycle, and a real fleet has both. The prefetch incumbent meanwhile ships regardless: vLLM v0.30.0's HiSparse host-resident tier spills KV pages to pinned host memory with per-request GPU hot buffers and batched host-to-device mediation — well-engineered prefetch-family machinery, exactly the paradigm BOOST's −6%/+17% numbers price⁵. The serving-engine churn context matters here: read engine-release features against the measured claims, not the release notes.

7. What we verified, what we did not, and the falsification list

What is load-bearing here, all checked at primaries for this guide: the abstract numbers (31%, 4.3%, −6%, 15%)¹³, the Grace Hopper numbers in the HTML full text (3.63 TiB/s spec, 3,330/350 GiB/s measured, α = 10%, 211 GiB/s prefetch loss, 211=6.3% of baseline, 9.6 GB host reserve, 800 LoC, vLLM v0.17.0, the 7.4% CAP peak and 6.7% oversubscription penalty, the batch-size crossovers, GB200 α=3%, Vera α≤44%)². The (1+α)/(1−α) ledger bound is the paper's own derivation and we recomputed it in Python above; our staging-buffer walkthrough (70 GB FP8 weights, K=10 → 5+ GB stale every step) is our model of their Fig. 3, using their measured loss as the anchor — treat the walkthrough as a sanity check, the 211 GiB/s as the ground truth.

Honest scope limits on the result itself:

1. One hardware family, one engine, one paper version. GH200 only, vLLM v0.17.0 only, v1 at submission (Sept 11, 2026)¹. No SGLang integration, no GB200 or H100 numbers except via α-emulation, no 2-GPU tensor-parallel scaling evidence beyond the Qwen2.5-72B study. Replicability optimism should be priced accordingly.
2. FP8 models throughout. The ledger holds at any precision, but the ratio of KV-to-weight bytes — and therefore how much of the win is attention- vs weight-driven — shifts with quantization; FP4 KV-heavy workloads are not directly measured.
3. Wave-awareness is API-fragile. MPP's determinism depends on 2 MB pages, current CTA structure, and current tiling. A driver update that changes page size or a kernel library switch that changes CTA shape degrades BOOST toward the random-placement distribution — the runtime chases a moving token, silently.
4. Peak-vs-measured gap. Spec-sheet arithmetic with the marketing numbers (which for some SKUs top 8 TB/s-class HBM3e) instead of the measured 3,330 GiB/s understates α and every derived number; never mix spec-sheet numerators with in-practice denominators — including this paper's own 419-vs-350 GiB/s C2C spread.
5. The UNISON lever, stated bluntly: BOOST's measured-on-silicon credibility cuts both ways — early framings of UNISON's validation-origin gap (a trace model, not Rubin silicon) reads harsher now. Watch the silicon path for hardware-measured turnaround first before pricing its other claims. That watchlist is yours to set up.

The verdict, sans hype: no revolution in silicon, no free 31%, one crisp accounting result that happens to arrive with the right hardware to demonstrate it on. Every tiering system that stages host data into fast memory during decode is consuming the very interface it is trying to protect; while NVLink-class coupling keeps host bandwidth within a generation of the fast tier, read both at once instead — in proportion, waves apart. The degree to which that translates to your fleet depends on the number nobody's press release contains: your α, measured, on your SKU, with your batch sizes.

Sources