HBF as the Third KV Tier: 24x Sessions or 5x Worse Latency - the Medium Is Fine, the Placement Policy Decides

Two papers, four weeks apart in the summer of 2026, looked at the same new memory technology — High-Bandwidth Flash (HBF), 3D-NAND stacked behind an HBM-style on-package interface — and reached opposite verdicts. A POSTECH-lineage IEEE CAL paper (arXiv:2609.25782, Baek, Ji, Yoo, Kim) reports that an HBM-plus-HBF hot-cold hierarchy hosts 24x more concurrent sessions per GPU while cutting read power by 7.6 kW per 8-GPU node¹. A Peking University full-stack characterization (arXiv:2608.11668, Li, Bian, Huang, Zhao, Sun, Zhuo) — titled "HBF Sucks?" — built the obvious system, an SSD-style Mooncake KV-offload stack with HBF swapped in underneath, and measured end-to-end latency 2–5.5x worse and maximum SLO goodput down 1.1–2.7x².

Neither paper is wrong, and that is the point of this guide. The medium is the same in both; the stream shape the medium is asked to absorb is not. One design hands HBF a small, immutable, written-once cold pool that is read only when a paused agent session resumes. The other hands it the relentless transient-KV write stream of a busy serving node — 48–140 TB of writes per day across the HBF-Sucks traces, every trace past both endurance envelopes — a workload that violates, in the counter-paper's own words, all three conditions a faster far tier needs to pay off: read-I/O-bound behavior, reads outnumbering writes, and sustainable delivered bandwidth². The deciding variable is placement policy, not silicon. This guide recomputes the arithmetic of both verdicts in Python, prices the flash medium honestly (endurance, latency, energy, page granularity), and slots HBF into the tiering thread we have been building all month: UNISON for host-DRAM scheduling of paused sessions, BOOST for concurrent host access, and now the on-package flash tier that both of those ultimately feed into.

1. What HBF actually is, and what is verified vs simulated

HBF applies HBM's packaging discipline to NAND: thinned dies stacked behind a base die, presented through a wide on-package interface. The first public specification, released by SanDisk and SK hynix through the Open Compute Project on August 3, 2026, pins down the systems contract: 8-high and 16-high NAND stacks reaching 512 GB per stack, three graded bandwidth classes spanning roughly 0.4 to 3.0 TB/s per stack, and UCIe as the host interface rather than a proprietary PHY³. SanDisk's first-generation targets are 1.6 TB/s read and 512 GB per stack — against HBM4's 48–64 GB per stack, that is roughly 8–10x the capacity per stack (SanDisk claims 8–16x capacity at a similar cost per stack, the sharpest sentence in their materials)³. The costs that come with the bits: reads complete in ~25 microseconds — two orders of magnitude behind HBM's ~100 ns — at page granularity of tens of kilobytes, and endurance is bounded by program/erase cycles that no amount of stacking changes.

Timeline honesty, because vendor slides will blur it: SanDisk's August 2025 roadmap promised HBF samples in the second half of 2026 and inference-device samples in early 2027. As of the August 13, 2026 Investor Day, the first HBF die has taped out, inference-product samples are still pegged at 2027, and mass production at 2028 — while SK hynix's FMS 2026 keynote put full-spec samples at early 2028⁴. Nobody outside the vendors has measured a physical stack. Every quantitative result below — including the paper this article centers on — is trace-driven simulation over analytic models, not silicon. The CAL paper's own methodology section says it plainly: an in-house, trace-driven simulator with analytical latency and power models on a modeled B200 node⁵. Label accordingly.

2. Bimodal agentic KV: why the access distribution has two humps

The CAL paper's first contribution is a workload observation rather than a hardware one, and it is the observation everything else depends on. Agentic sessions do not behave like chat traffic. A session lives for hours as a loop of decode steps, tool calls, and user inputs; its KV cache accumulates across the entire session instead of resetting per query. While it decodes, the whole cache is read every step. While it waits on a tool or a human — which is most of the time; the paper models idle gaps with a median of ~8 seconds and a mean of ~23 seconds — the cache is dead weight that still must be retained, because resuming without it means recomputing the entire context⁵.

We quantified this shape from the fleet side in our agent-fleet economics piece: on the Matt Barrie anchor day (~44 agents, ~4B tokens), roughly 75% of all tokens were cache reads of the fleet's own context, not fresh work — an agent that has run 50 steps re-sends its first 49 steps, 50 times. That looping fraction is the traffic-side signature of the same bimodality the CAL paper measures on the memory side: its simulation shows the hot set — the KV of the currently-decoding batch — occupying ~96 GiB, about 3% of total KV capacity, while drawing ~98% of read traffic⁵.

Run the session arithmetic for the paper's workload, Qwen3-Coder-30B-A3B (48 layers, 4 KV heads, head_dim 128 in our glossary's formula — 2 x layers x kv_heads x head_dim x 2 bytes per token, which reconstructs the paper's 96 KiB/token exactly):

kv_per_token = 2 * 48 * 4 * 128 * 2          # bytes, BF16
print(kv_per_token / 1024)                    # 96.0 KiB -> matches paper
peak_ctx = 16700
print(peak_ctx * kv_per_token / 2**30)         # 1.53 GiB KV at peak 16.7k context
print(3900 / 165)                              # 23.6 -> the abstract's "24x"

An HBM-only 192 GiB B200 saturates at 165 concurrent sessions for this trace; adding a 3 TiB HBF cold tier expands it to 3,900⁵. Both numbers are simulation outputs, and the "24x" is a rounding of 23.6x — but the capacity mechanism is just arithmetic: the cold pool and the hot set compete for the same 192 GiB without tiering, and the cold pool is ~30x the hot set.

3. Write-on-evict: the placement policy that keeps endurance inside warranty

The CAL paper's policy insight is a single sentence worth memorizing: because autoregressive decoding only appends and a KV block is never modified after it is written, KV blocks are immutable — so under a write-on-evict policy, each block is written to HBF at most once in its lifetime. The block reaches HBF only when demoted from HBM; if it is re-admitted on resume and evicted again, the HBF copy is still valid, so no rewrite happens⁵. Writes are confined to the eviction path entirely, and eviction is by definition an idle-block event: LRU naturally keeps the per-step-read hot set in HBM and demotes paused-session state that nothing is touching.

That is what makes flash endurance a non-story in this design and a headline in the SSD-offload one. Do the budget arithmetic for the paper's SLC assumption (3 TiB, 100,000 P/E cycles, write amplification 1.02 thanks to erase-block-aligned immutable appends — versus 2–4 for a general-purpose SSD):

budget_gib = 3 * 1024 * 100_000               # total writable GiB over the stack's life (3 TiB x 100k P/E)
churn_day = 3900 * 0.82                       # GiB if the whole cold pool turned once/day
print(budget_gib, "GiB budget;", churn_day, "GiB/day full-churn")
print(budget_gib / churn_day / 365)   # ~263 years at naive full churn

The naive bound is absurdly safe, which is why the paper's own figure is the load-dependent one: write-on-evict sits at effectively unbounded lifetime below a decode-batch size of 32 (nothing gets evicted) and still reaches ~20 years at full load, while all-KV-to-flash — the policy that commits every produced KV block to HBF — falls from ~11 to ~8 years and only clears the 5-year warranty line⁵. The same stack, same media, same endurance rating: at least once written versus written every step. Placement policy is the entire difference.

Now the counter-paper's side of the same ledger. HBF-Sucks measures real production traces (four two-hour Qwen-Bailian traces, five dense and MoE models) and finds transient KV generates 48–140 TB of writes per day on a busy node, every trace past both endurance envelopes². Put that against a TLC HBF stack — TLC is what a cost-optimized all-flash pool would actually use — and its ~3,000 P/E rating:

tlc_budget_gib = 3 * 1024 * 3000  # GiB a 3-TiB TLC package can write in its life
churn_gib = 140e12 / 2**30 / 8    # 140 TB/day node churn, spread over 8 packages
print(tlc_budget_gib / churn_gib / 365)          # ~1.5 years per package
print(tlc_budget_gib / (140e12 / 2**30) / 365)   # ~0.19 years if one package took it all

That is the HBF-Sucks endurance verdict in a few lines of Python, and it is not a criticism of NAND — the capacity-matched SSD pool it replaced (four KIOXIA CM7-V 3.2 TB, 38.4 TB/day vendor rating vs the HBF tier's 21.7 TB/day budget) spreads the same writes over far more raw NAND. It is a criticism of routing a write-heavy, low-reuse stream into a tier whose write budget is bounded by on-package capacity.

4. Why the drop-in fails: three violated conditions and one brutal misdirection

The HBF-Sucks paper's most quotetable finding is its most misread one. When the authors scaled HBF's read and write latency 3.75x, end-to-end latency moved less than 1%². Flash latency — the thing everyone instinctively blames — is nearly irrelevant here. What kills the drop-in is structural: the two-tier Mooncake-style hierarchy keeps reusable KV in the near tier and hands HBF "a relentless write-heavy stream. Writes outnumber reads on every trace," so a 3D-ICE thermal model shows the stack hitting its thermal limit well below peak bandwidth². The faster device yields a slower system because it is being used as a dump: transient KV with near-zero reuse is written, evicted, and overwritten at the write side of a medium whose economic value is all on the read side.

Hence the cost-benefit model the paper extracts: a faster far tier pays off only when (1) read I/O is the bottleneck, (2) reads outweigh writes, and (3) delivered bandwidth is sustainable. Transient KV violates all three simultaneously². Paused-session cold pools violate none — they are written once, read once per resume, and idle in between. Same medium, opposite verdicts, and the deciding variable is which stream you point at it.

5. The 25-microsecond question: resume path fine, per-step path fatal

The CAL paper models HBF read access latency at 25 microseconds. Is that fast or slow? It depends entirely on the frequency with which the read occurs — do the latency-budget arithmetic against the decode step it would sit on:

tbt_ms = 14          # paper's TBT at 72 tok/s per agent, A within SLO
hbf_us = 25
print(hbf_us / (tbt_ms * 1000))     # 0.18% of one decode step per read
 
hot_gib = 96         # hot set at the simulated operating point
for bw_tbps in (0.4, 3.0):          # HBF spec Grades 1 and 3
    print(f"hot set from HBF: {hot_gib*1024/bw_tbps/1000:.0f} ms/step vs 50 ms SLO")
# Grade 1: 246 ms/step; Grade 3: 33 ms/step -> both over budget with compute on top

Once per resume, amortized across a sub-array-parallel fetch of the resumed session's KV and overlapped with the CPU-side work (tokenization, results parsing) that precedes every agent turn, 25 microseconds per block is noise: the paper attributes ~0.084 ms of overhead to an HBF-backed resume, against 1.0 ms over NVLink-C2C host memory, 7 ms over PCIe, and 14 ms of recompute — where recompute doubles the turn cost at high concurrency and HBF fetches are pure I/O off the compute path⁵. Note the ordering: at flash-class bandwidth, on-package HBF beats a host-DRAM offload tier on resume — the same interconnect-bound argument as our BOOST piece, arrived at from the cold-pool side.

Once per decode step, the same latency is a wall: the hot set must stream from a tier that sustains it every 14 ms, which is why the paper's design has decode reading KV from HBM, always — resume restores to HBM first, so TBT is independent of the cold tier entirely⁵. This is exactly the 25-microsecond nuance vendor decks will blur: HBF latency is fine for an idle-fanout resume path and disqualifying for a batch-synchronous per-step tier. The paper that puts HBF on the per-step path for transient KV — HBFlex, serving entirely from HBF on a DeepSeek-V4-Pro workload — has to spend its whole architecture budget (placement balancing, aggregated writeback in compute windows, lifetime-guided block packing) fighting write-read interference and garbage collection that reaches a write amplification of ~30x, and it too is trace-driven simulation⁶. The gc-number is the SSD-offload pathology wearing a different hat.

6. The -7.6 kW power claim, recomputed

The abstract's second headline — writes-on-evict cuts read power by 7.6 kW per 8-GPU node relative to serving all KV from flash — sounds like a green-marketing number, but it is the read-energy gap doing mechanical work. HBF's per-bit read energy is unstandardized (the paper sweeps 8–30 pJ/bit); HBM sits at ~3.5 pJ/bit. At the modeled operating point — hot set 96 GiB, read every 14 ms decode step, assuming 20 pJ/bit — the all-KV-to-flash baseline pays HBF read energy on every actively read KV byte:

hot, step_s = 96, 0.014
rate = hot * 2**30 / step_s        # bytes/s of hot-set reads
p_hbf = rate * 20e-12 * 8          # W at 20 pJ/bit
p_hbm = rate * 3.5e-12 * 8
print(p_hbf, p_hbm, p_hbf - p_hbm)      # ~1178 W vs ~206 W -> ~972 W
print((p_hbf - p_hbm) * 8 / 1000)       # ~7.8 kW/node vs paper's 7.6 kW

Our reconstruction lands at 972 W per device against the paper's ~950 W and 7.8 kW against 7.6 kW per node — within modeling-rounding distance of their figure⁵ (the paper's sensitivity range across the swept read-energy span is 2.1–12.2 kW per node). The claim's direction is physically sound: NAND's advantage is idle power — the cold pool costs milliwatts per GB to hold, whereas the hot set's cost is paid on access, and restricting access to HBM confines the expensive joules to 3% of the bytes. But keep the caveat: this is a model output, not a meter reading. The 20 pJ/bit input is an assumption from a power-budget citation, swept precisely because no standard value exists⁵. Treat the kilowatts as a plausible range, not a datasheet.

7. Capacity economics: three populations, one sizing ledger

Why bother with on-package flash at all when host DRAM (UNISON's tier) and SSDs (Mooncake's tier) already exist and are shipping? Density per dollar. Flash delivers roughly 8–16x HBM's capacity at a similar cost per stack³; host DRAM adds a CPU, a coherent link, and someone else's memory margin; SSDs add milliseconds and a PCIe queue. For fleet sizing, model the KV population as three buckets and let each medium price its bucket:

• Hot (actively decoding, read every step): HBM. Bandwidth-bound, energy-bound, ~3% of KV bytes, ~98% of reads⁵.
• Paused (between turns, read once on resume): HBF. Written once on evict, read once on resume, ~25 microseconds of amortized latency against a multi-second idle gap.
• Evicted/terminated (session over, prefix maybe reusable later): SSD pool. The Mooncake reuse machinery earns its keep here — on truly cold data.

This is the three-layer division of labor the vendor-side analysis converged on as well: HBM for active KV and write-heavy state, HBF for large read-mostly objects, SSD for the cold remainder³ — and it is the capacity economics behind our HBM4-shortage piece: the cheapest way out of an HBM capacity crunch is to stop storing dead context in it. The general KV-tiering principle from earlier in this series holds at the new tier: the hierarchy should mirror the access distribution, not flatten it.

8. What is load-bearing, what is simulated, and the falsification list

Verified at primaries for this guide: the CAL paper's full text (165 -> 3,900 sessions, 96 KiB/token, 1.53 GiB peak KV, 0.82 GiB average resident, 14 ms TBT / 72 tok/s at operating point, 0.084 ms HBF vs 1.0 ms C2C vs 7 ms PCIe vs 14 ms recompute resume overheads, hot set 96 GiB / 3% capacity / 98% reads, write-on-evict ~20 years vs all-KV 8–11 years at load, 950 W/device and 2.1–12.2 kW/node across the 8–30 pJ/bit sweep, SLC 3 TiB / 100k P/E / WAF 1.02, idle gaps median ~8 s mean ~23 s)¹⁵; the HBF-Sucks abstract and model (2–5.5x latency, 1.1–2.7x goodput loss, 3.75x latency scaling moving E2E under 1%, the three-condition benefit model, thermal-limited bandwidth, TLC wearing out sooner than the SSD pool)²; HBFlex's measured-GC framing (write amp ~30x on DeepSeek-V4-Pro, also trace-simulated)⁶; the OCP specification details and vendor timelines³⁴. Our Python recomputes above: the 96 KiB/token reconstruction, the 1.53 GiB session size, the 23.6x session scaling, the 972 W -> 7.8 kW power reconstruction, the Grade-1/Grade-3 per-step budget, and the TLC endurance bound (~1.5 years per package at node-wide churn). Everything labeled simulation is simulation — no HBF silicon has been independently measured anywhere yet.

Falsification watchlist, in the order that would kill the thesis:

1. Physical stacks ship slower than the spec. All of this assumes Grade-1-to-3 bandwidth classes and ~25 microseconds hold in volume production. First independent measurements (2028-class, per current timelines⁴) move every number here: if effective bandwidth lands at the low grade and page-granularity penalties bite random resume fetches, the resume advantage over NVLink-C2C narrows to zero.
2. Real agent fleets may not be this paused. The bimodality rests on heavy-tailed idle gaps from a scaled replay — thousands of sessions reconstructed from a small SWE-bench trajectory set⁵. A fleet with short, dense tool loops and high duty cycle looks more like a throughput workload, where the paper concedes ordinary batch inference is better served by big-batch serving on HBM anyway.
3. The write-on-evict/LRU contract is fragile. Recency-as-policy works until the hot set itself overflows HBM (the paper avoids pinning precisely because pinning deadlocks there⁵). A workload with a large actively-read working set — long-context single-stream reasoning, not agent fanout — evicts its own hot set and pays 25 microseconds per step. The policy that saves you in one regime degrades you in the other: this is the same lesson as UNISON's duty-cycle modeling, one tier down.
4. The endurance figures are SLC. A cost-optimized product ships TLC or QLC; the HBF-Sucks TLC arithmetic above is then the relevant one, and write budgeting becomes a first-class software requirement — exactly the "write budgeting, reuse-aware placement, thermal coordination" the counter-paper prescribes².

9. Verdict, and where this lands in the thread

Sans hype: HBF is neither a revolution nor a mistake. It is a capacity tier with flash economics and near-memory bandwidth, and the two September papers are two halves of one engineering rule: give the flash the writes-never, reads-rarely population and it is the best capacity-per-watt-and-dollar tier in the stack; give it the writes-constantly, reads-rarely population and it is the worst. The 24x session claim and the 2–5.5x latency claim are both real, both simulated, and both about policies, not media.

In the thread: UNISON schedules paused-session KV across host DRAM and proved the scheduling problem is real; BOOST proved the host tier is a bandwidth peer during decode when accessed concurrently; the CAL paper shows why the next tier up — on-package flash — makes the UNISON-style cold-pool problem easier (resume in 0.084 ms instead of 1.0 ms over C2C) but does not make it disappear: something still has to decide which population lives where and when. That decision layer is precisely where UNISON's scheduling silicon, BOOST's proportional placement, and HBF's write-on-evict policy meet. If the 2027 HBF hardware lands anywhere near spec, the 2028 argument in this series will not be whether to add the tier — it will be who owns the placement policy that feeds it. The flash does not care. The stream shape is everything.

Sources