CXL-SSDs For LLM Prefix Caching: Byte-Addressability Buys Nothing Without Chunk Awareness

A September 2026 systems paper opens with a result that should temper a lot of CXL marketing decks: when you plug an unmodified, generic CXL-SSD into an LLM prefix-caching stack, it is about 3× slower than local DRAM and no faster than a plain NVMe SSD — and the standard sequential prefetcher you would reach for to fix that leaves average time-to-first-token (TTFT) nearly unchanged while performing irrelevant NAND reads¹. The paper, Bridging LLM Serving and CXL-SSDs with Chunk-Aware KV Cache Management (Chung, Noh, Kim and colleagues at Sogang University with Samsung Electronics and ETRI, arXiv:2609.26828, cs.AR, submitted September 20, 2026), then earns its constructive half: LM-CXD, a CXL-SSD specialized for LLM prefix caching that closes the semantic gap between the serving engine (which knows which KV chunks will be consumed) and the device (which controls their placement and movement), cutting average TTFT by up to 2.6× over the stock CXL-SSD with compute-asynchronous prefetching and 4.03× with layerwise prefetching — landing within 1.5× of local DRAM on average¹.

The negative result is the load-bearing part, because it cleanly separates two claims that CXL marketing perennially fuses. Claim one: the block I/O path is expensive for KV retrieval regardless of the medium beneath it — CPU cache contention, host-DRAM staging copies, syscall bookkeeping. Claim two: CXL's byte-addressable, memory-semantic interface removes those costs. The paper proves claim one with a beautiful experiment (a ramdisk that keeps DRAM behind the block interface remains 1.8× slower than direct DRAM, with no NAND anywhere in the system) and then refutes the naive reading of claim two: removing the block path does not make a CXL-SSD faster than an NVMe SSD for this workload, because the costs that remain — NAND latency and device-side thrash in a bounded DRAM region — are not addressed by addressability at all¹.

This guide walks the full stack in arithmetic: the interface-cost decomposition, the silicon reality of CXL versus PCIe bandwidth, why chunk granularities and NAND granularities collide, why any prefetcher without engine knowledge must speculate and lose, and the layerwise pipelining bound that decides whether NAND latency actually hides under compute. Every number cited from the paper was verified against the full HTML text, not the abstract².

1. The problem: prefix caching has outgrown DRAM, and the block path is why SSDs hurt

Modern LLM traffic — RAG contexts, multi-turn chat, agentic loops — re-sends prefixes of tens to hundreds of thousands of tokens. Prefix caching reuses the KV caches computed for those prefixes to skip redundant prefill, and systems like LMCache extend the cache beyond GPU memory into host DRAM and storage³. But host DRAM is costly and bounded by memory channels and DIMM capacity, so production stacks drop the cold tier onto NAND-backed NVMe SSDs. NAND exposes that capacity only through the block interface, and the paper's characterization of a prefix-reuse workload (Qwen3-4B on vLLM with LMCache, 128k prefix, 32-token chunks, four storage configurations) traces the degraded TTFT to precisely three costs¹:

1. CPU cache contention. Block I/O runs on host threads that share the last-level cache (LLC) with inference threads, using it for entirely different purposes. Isolating the I/O threads' L3 partition with Intel Cache Allocation Technology (CAT) recovers up to 44% of P99 TTFT — a large, tail-shaped cost for something as mundane as syscalls touching page cache⁴.
2. Host-DRAM staging. The block interface forces every chunk through host DRAM before the GPU can read it, because the GPU can DMA only from pinned pages mapped into the application's address space. Measured with VTune: on a ramdisk, writes rise from 6% to 33% of DRAM traffic, holding DRAM busy 3.6× as long per retrieval. A pinned-DRAM baseline at 1.34 s average TTFT degrades to 5.27 s on ext4-over-brd, and stripping ext4 recovers only 0.2 s of the 3.9 s gap — the staging copy is the filesystem-independent part¹.
3. NAND latency itself. Replacing NAND with DRAM (the ramdisk) closes 61% of the SSD's TTFT gap to direct DRAM in steady state — so a bit over half the SSD slowdown is the medium, and the rest is the interface.

The first two costs are properties of the interface, not the medium. That is the paper's first durable finding: a ramdisk — pure DRAM, served through the block path — is still 1.8× slower than direct DRAM. And the obvious escape hatches fail for structural reasons. Host-side prefetch overlaps NAND latency but reintroduces the DRAM-capacity tension: you use SSDs because host DRAM is limited, yet prefetched chunks must occupy host DRAM before consumption. GPUDirect Storage (GDS) bypasses CPU and host DRAM but does not remove the block-oriented path: cuFileRead places data in a VRAM staging buffer, after which load_and_reshape reformats and copies it into vLLM’s paged KV layout. The paper reports that this serialized extra GPU-side step leaves a performance gap; it is one staging-to-final-layout copy, not two⁵.

2. Why byte-addressable does not mean faster: the silicon underneath

CXL is a cache-coherent interconnect built on PCIe physical infrastructure. A CXL Type-3 device exposes Host-managed Device Memory (HDM) through the CXL.mem protocol, mapped into the host physical address space and accessed with ordinary load/store instructions⁶. A CXL-SSD bolts that onto an SSD architecture: controller, on-device DRAM (the DRAM region, serving hits at memory latency), and NAND flash behind a flash translation layer¹.

Here is the arithmetic the marketing glosses over. A PCIe 5.0 x16 link carries roughly 64 GB/s raw in each direction; after 128b/130b encoding, packet overhead, and the fact that a CXL.mem read is a round trip (request plus data, plus the dHDM coherency machinery), effective sustained bandwidth for memory-semantic loads is below the link’s headline rate and depends on the device, topology and transaction pattern. An aggregate host DRAM system on a modern dual-socket server delivers several times that across its channels. So the ceiling of CXL-attached memory is not DRAM speed; it is PCIe speed — the same wall NVMe sits behind. And below the DRAM region, a CXL-SSD miss takes the CXL.mem path to its controller, then FTL lookup and NAND read latency; the paper models Z-NAND with tR = 3 µs¹.

That is why "byte-addressable" is a programming-model property, not a performance property:

• The DMA toll is the same link either way. Whether the GPU's source buffer is pinned host DRAM or a cudaHostRegister-mapped CXL region, the host-to-GPU transfer traverses the same PCIe hierarchy. Addressability removes the staging copy, not the transfer.
• Cache-line traffic still crosses the link. A random cache-line pull from device DRAM costs a full CXL.mem transaction pair (request line, return line, with coherency state). Bursty KV fills at 2 MiB hugepage granularity amortize this fine; a miss-driven, pointer-chasing retrieval pattern does not.
• Contention doesn't vanish, it relocates. With the block path gone, I/O threads disappear — but under load, every CXL-SSD read still competes for the shared link, the device's on-device DRAM region bandwidth, and the FTL's NAND channels.

The next cell prices the whole retrieval path for a realistic single-request retrieval — 18 GiB of KV (a 128k-token prefix for Qwen3-4B: 36 layers, GQA with 8 KV heads of 128 dimensions, BF16, in 56-token chunks) — under each backend, both as wall time and as delivered GB/s. The bandwidth values are assumed, stylized rates, not measurements in the paper. The paper’s initial characterization uses 32-token chunks; its later Qwen3-4B evaluation uses 56-token chunks¹.

import math
 
# Interface-cost arithmetic for one 128k-token KV retrieval (Qwen3-4B shape:
# GQA 8 KV heads x 128 head dim, BF16, 36 layers, 56-token LMCache chunks).
# Stylized bandwidths; ratios, not absolute times, are the point.
TOK, LAYERS, CHUNK_TOK = 131072, 36, 56
KV_TOK_LAYER = 2 * 8 * 128 * 2                 # K+V bytes per token per layer
CHUNK_B  = CHUNK_TOK * KV_TOK_LAYER * LAYERS   # bytes per KV chunk
NCHUNK   = math.ceil(TOK / CHUNK_TOK)  # final chunk can be partial
PAYLOAD  = TOK * KV_TOK_LAYER * LAYERS
 
B_HOST, B_PCIE = 40e9, 20e9    # host DRAM rw, host<->GPU link (every path pays this)
B_NVME, B_GDS  = 7e9, 12e9     # NVMe steady read, GDS NVMe->VRAM DMA
B_VRAM         = 40e9          # VRAM-internal copy bandwidth (load_and_reshape)
SYS_US         = 15e-6         # per-chunk block-path kernel/bookkeeping charge
 
base = PAYLOAD / B_PCIE        # the DMA toll no path avoids
rows = [
 ("pinned host DRAM",      [(1, B_PCIE)]),
 ("ramdisk (brd+ext4)",   [(1, B_HOST), (1, B_PCIE)]),
 ("local NVMe O_DIRECT",   [(1, B_NVME), (1, B_PCIE)]),
 ("GDS NVMe -> VRAM",      [(1, B_GDS), (1.35, B_VRAM)]),
 ("CXL-SSD, DRAM hit",    [(1, B_PCIE)]),
 ("CXL-SSD, NAND miss",   [(1, B_NVME), (1, B_PCIE)]),
]
print(f"payload {PAYLOAD/2**30:.2f} GiB in {NCHUNK} chunks (up to {CHUNK_B/2**20:.2f} MiB each) "
      f"(multi-hugepage: {CHUNK_B/2**20 > 2})")
print(f"{'path':26s} {'stages':>6s} {'wall ms':>9s} {'deliv GB/s':>10s} {'vs pinned':>9s}")
for name, stages in rows:
    t = sum(PAYLOAD * f / b for f, b in stages) + SYS_US * (NCHUNK if "NVMe" in name or "ramdisk" in name else 0)
    print(f"{name:26s} {len(stages):5d}  {t*1000:9.1f} {PAYLOAD/t/1e9:9.2f} {t/base:9.2f}x")
 
# Why byte-addressable != faster, mechanically:
# 1) the DMA toll is the same PCIe link for pinned DRAM and CXL-mapped memory;
# 2) block paths add host-DRAM staging passes (writes 6% -> 33% of DRAM traffic
#    in the paper, DRAM busy 3.6x as long per retrieval);
# 3) I/O shares the LLC -- isolating the I/O partition with Intel CAT recovers
#    up to 44% of P99 TTFT in the paper. Overlapped degradation, stylized:
print("\nretrieval overlapped with decode, LLC partitioning off (stylized):")
for drop in (0.30, 0.44):
    t = PAYLOAD / (B_PCIE * (1 - drop))
    print(f"  effective DMA -{int(drop*100)}% -> {PAYLOAD/t/1e9:5.2f} GB/s ({t/base:4.2f}x slow-down)")

Output from a real run (Python 3, seed 260926828):

payload 18.00 GiB in 2341 chunks (up to 7.88 MiB each) (multi-hugepage: True)
path                       stages   wall ms deliv GB/s vs pinned
pinned host DRAM               1      966.4     20.00      1.00x
ramdisk (brd+ext4)             2     1484.7     13.02      1.54x
local NVMe O_DIRECT            2     3762.5      5.14      3.89x
GDS NVMe -> VRAM               2     2298.0      8.41      2.38x
CXL-SSD, DRAM hit              1      966.4     20.00      1.00x
CXL-SSD, NAND miss             2     3727.4      5.19      3.86x
 
retrieval overlapped with decode, LLC partitioning off (stylized):
  effective DMA -30% -> 14.00 GB/s (1.43x slow-down)
  effective DMA -44% -> 11.20 GB/s (1.79x slow-down)

These are illustrative serial-transfer calculations, not replicas of the paper’s TTFT measurements. The chosen rates make the ramdisk 1.54× slower than pinned DRAM and NVMe 3.89× slower; neither figure validates the paper’s measured 1.8× and roughly 3× comparisons. The toy assigns CXL-DRAM hits the same transfer rate as pinned DRAM and CXL-NAND misses the same rate as NVMe by assumption. Real hit and miss time also depends on CXL transactions, device-controller behavior, concurrency and overlap. The paper’s measured result is that its stock-device setup failed to beat NVMe, while chunk-aware coordination improved TTFT¹.

3. The CXL-SSD paradox: a better interface, no better TTFT

The paper's § 4.1 experiment is the one to commit to memory. They run the multi-turn Q&A workload against a stock CXL-SSD implementation — same 2 MiB hugepages and CLOCK eviction the paper's own device will use, LMCache asynchronous loading enabled, no device-level prefetch — and against the same device with Cylon's standard next-n sequential prefetch policy, n = 4 pages per DRAM miss⁷. Results: applying the CXL-SSD without specialization does not improve performance over the SSD baseline; P99 TTFT regresses above the SSD's; and next-n leaves average TTFT nearly unchanged, with some turns worse. The counter comparison is precise: next-n issues 19,198 async loads, but evictions rise from 369,104 to 387,840 and NAND reads from 1,494,131 to 1,504,029 — prefetch added NAND traffic and evictions instead of saving them¹.

Why does a prefetcher make things worse? Because all CXL-SSD I/O must go through the bounded DRAM region, and a stock device's placement policy has no idea what is coming. Sixteen concurrent sessions sharing a 16 GiB region while each needs GiB-scale prefixes means thrash: chunks repeatedly evicted and re-fetched between DRAM and NAND. Next-n cannot help because it is miss-driven and speculative: it must first incur a DRAM miss to trigger, guesses the next sequential pages by address, and when its guess is late or wrong it pays an extra miss and burns NAND bandwidth on pages nobody wanted. The vicious cycle the paper names is structural: miss-driven prefetch must incur the miss it is trying to prevent¹.

And here is the granularity collision that makes speculation hopeless. LMCache's unit is the chunk — 32 to 128 tokens depending on the model, roughly 8–40 MiB of KV bytes at these shapes — while the device's unit is the 2 MiB hugepage, itself an alignment imposed on 4 KiB OS pages, mapping down onto NAND pages (tens of KiB) grouped into erase blocks (typically hundreds of KiB to a few MiB). A layerwise-inference read wants one layer across many chunks, which is neither page-shaped nor chunk-shaped. A device that sees only page addresses cannot serve either access shape, and it certainly cannot predict it: KV reuse follows the tree structure of prefixes — system prompt, retrieved documents, conversation history — not the address layout of a flat array. The serving engine knows the shape of that tree at prefix-lookup time, before any byte moves. The device never sees it. That asymmetry — not the absence of prefetching — is the gap.

The next cell reproduces the DRAThe measured counters already show the mechanism without a synthetic cache trace: the paper's next-‑n configuration issues 19,198 asynchronous prefetch loads, yet total NAND reads rise by 9,898 and evictions by 18,736 relative to no prefetch. Those figures establish extra traffic in this benchmark; they do not identify how many prefetched pages were never consumed or establish a general hit-rate change. LM-CXD instead exposes the known KV chunks and progress to the device and serving engine, then pins bounded windows while they are needed¹.

ign, one property at a time

The paper's move is to stop treating the CXL-SSD as a generic memory device and make the KV chunk the unit the device indexes, moves, reports progress on, and transfers to the GPU. Three properties, each closing a specific measured failure¹:

Property 1: chunk identity becomes I/O. LMCache's prefix lookup already computes chunk hashes for every request at arrival — that is how it finds the longest cached prefix. LM-CXD's shared structures (prefix lookup table tracking chunk-hash to location, a pin map protecting in-flight windows from CLOCK eviction, a free queue for chunk deletion, a prefetch queue of chunk hashes, and a prefetch plan pool for layerwise) live in a reserved region of device DRAM, readable and writable by both LMCache and the device controller with no syscalls, interrupts, or copies. LM-CXD resolves chunk hashes against its own index, skips chunks already in the DRAM region, and issues NAND reads itself — the cost of chunk-matching on the host becomes 6–7 microseconds per chunk under compute-asynchronous prefetching (CAP), against the 160 microseconds per chunk the stock device spends copying¹.

Property 2: the engine acts on device progress. LM-CXD records per-session dispatch and completion state in device DRAM. LMCache polls it, admits a session once its first two windows' worth of chunks are dispatched (not completed), and DMAs finished chunks to the GPU while the session waits in vLLM's non-blocking WAITING_FOR_REMOTE_KVS state, under other sessions' compute. This is the bidirectional half: hints flow engine-to-device, progress flows device-to-engine, and admission stops waiting for completion of loads it can overlap. Promotion is capped per scheduling step with one unconditional promotion, so the dispatcher drains between admissions.

Property 3: device DRAM is the GPU's source buffer. The DRAM region is cudaHostRegister-mapped pinned memory; DMA pulls chunks straight from device DRAM to VRAM without host staging. The windowing discipline (a per-rank pin budget sized to the region, round count, and workload; primed two windows ahead; the next window starts only when another unpins) is what makes PREFETCH possible in a region that cannot hold an entire prefix — the paper's third design principle, hide NAND latency under bounded device DRAM, made concrete.

Then two prefetch methods, each matched to a workload regime. CAP paces windowed prefetch behind lookup hints while other sessions compute — the benchmark answer for concurrent multi-turn serving. Layerwise prefetching goes further: it tracks a per-request plan, converts chunks from chunk-major to layer-major layout in a dedicated staging area of device DRAM, and uses a strided cudaMemcpy3DAsync descriptor so the DMA itself expresses the layout change (two copies per layer instead of two per chunk). A constant n-deep lookahead buffer lets NAND reads run ahead of layer consumption; the paper runs prefetch degree 7 on Qwen3-VL-32B.

5. When layerwise prefetch helps: the measured queue and prefill tradeoff

Layerwise retrieval starts NAND reads ahead of each layer’s consumption. Whether this helps depends on the available compute overlap, the number of concurrent sessions and the device-DRAM buffers reserved for lookahead. A per-layer FLOP estimate alone cannot predict the result: it omits the actual kernels, scheduling, cache hits, contention and queue admission. Table 5 of the paper provides the relevant measured lifecycle times instead¹:

Layerwise has no separate admission queue in this comparison: it folds retrieval into prefill. For Qwen3-4B at 64k, that adds 1.40 s of prefill while removing only 0.51 s of CAP queue time, so CAP wins in this workload. For Qwen3-VL-32B, the 0.30 s extra prefill is smaller than CAP’s 1.06 s queue; layerwise wins. Llama’s small 0.02 s mean prefill difference does not imply a safe latency tail: at 16k its P99 NAND wait remains within 1.3× of the stock device, because a late layer has little compute in which to hide its read¹.

Lookahead has a capacity price even when it hides latency. The paper reports about 9 GiB of layerwise buffers for either Qwen model at 64k prefixes, displacing over half of the 16 GiB device-DRAM chunk region; Llama at 16k uses 2.5 GiB. The results are specific to the evaluated models, prefix lengths and concurrency. They do not establish an optimal lookahead degree for other deployments¹.

6. What the co-design actually buys, measured

Across five models (Qwen3-4B, Ministral-8B, Qwen3-Coder-30B-A3B, Qwen3-VL-32B, Llama-3.1-70B — all GQA) on the CxS multi-turn benchmark (16 sessions, 6 turns) and an agentic-coding trace (AIPerf, 48 sessions, 128k–160k prefixes): TTFT falls 2.51× and 2.6× on average under CAP for Qwen3-4B and Ministral-8B; 4.03× for Qwen3-VL-32B under layerwise; 1.74× for Llama-70B; within 1.5× of local DRAM on average overall¹. Median felt-NAND time per request falls from 139–920 ms (stock, across the four models) by at least 3× at the median and 2.5–4.1× at P90 under LM-CXD — with honest hundreds-of-milliseconds tails remaining. On the agentic trace, mean TTFT is nearly identical to stock (most of that KV stays hot in the DRAM region by reuse frequency) and the win is a 17% lower P90 — chunk-aware prefetch pays exactly where reuse is sporadic, and nowhere else.

The DRAM-region sensitivity table is the cleanest pricing of the semantic gap: at 8/16/32 GiB regions, the stock CXL-SSD averages 6.26/6.53/1.98 s TTFT while LM-CXD holds 2.78/2.40/1.90 s. The stock device is highly region-size-sensitive (3× swing); LM-CXD moves less than 1.5× across the sweep and at 8 GiB beats the stock device at 16 GiB by 2.3×. The lifecycle split shows why: the stock device's queue and prefill times both rise about threefold at small regions, while LM-CXD admits on dispatch and copies pinned windows, so its growth lands in the queue alone. The shared-structure overhead is negligible in the other direction — 22 MiB, 0.13% of a 16 GiB region, for everything CAP needs¹.

7. Anti-hype: what this paper does not establish

The CXL-SSD results use an emulated device. No CXL-SSD was commercially available for this evaluation; the host GPUs and serving software are real, while LM-CXD is implemented on a modified NVMeVirt platform (page-fault-intercepted DRAM with an injected Z-NAND timing model, NUMA-node memory standing in for CXL timings), validated against the Cylon emulator on bimodal latency and per-miss breakdown (matched ≈6.2 microsecond device time; the residual difference is Cylon's KVM/QEMU exit overhead, not device physics)⁷. Every absolute number inherits emulation error; the mechanism claims are the transferable part.

The observed 1.5× average gap to DRAM has known causes. Even fully specialized, device DRAM is bounded, NAND reads remain on the tail, and LM-CXD's queue-time component does not vanish — at 32 GiB regions the paper reports the two devices converge and LM-CXD's prefill matches local DRAM's, leaving queue time as the residual. Chunk-aware design narrows the gap; it does not close it.

Co-design cuts both ways. LM-CXD's bidirectional interface costs host-side bookkeeping (1.2–2.4% of a retrieve — small but nonzero), the lookahead buffers displace more than half the chunk region at 64k prefixes, and layerwise is a net loss for small-prefix, fast-decode models (Qwen3-4B: +1.40 s prefill to remove 0.51 s of queue wait). The paper's own cross-model comparison shows no single prefetch method winning everywhere — CAP for concurrent serving, layerwise for large-single-session loads — which is the standard signature of a real system rather than a benchmark artifact.

KV cache regenerability is load-bearing. LM-CXD discards live chunks from erased NAND lines rather than relocating them — no garbage collection — because KV can always be recomputed. That is a prefix-cache-specific exemption; the design does not generalize to storage where data must survive erasures¹.

Byte-addressability helps this design by avoiding block staging. In the paper’s measured setup, putting DRAM behind a block interface cost 1.8× relative to direct DRAM access. LM-CXD removes that path and adds chunk-aware scheduling; the paper does not prove byte-addressability is necessary for every possible storage design or that device-DRAM hits always match local DRAM. Its 2.6–4.03× gains compare complete LM-CXD configurations with the stock CXL-SSD, so they cannot be attributed to one mechanism alone.

For a practitioner the checklist distills to: (1) keep hot prefixes in plain DRAM first — the stock CXL-SSD loses to local DRAM by 3×, and even LM-CXD concedes 1.5×; (2) if a NAND tier is unavoidable and CXL-SSDs materialize, nothing generic helps — demand chunk-visible interfaces, windowed pinned prefetch, dispatch-progressive admission, and layerwise conversion; (3) budget the device DRAM region explicitly and check the per-layer transfer-versus-compute ratio at your prefix lengths before promising anyone 4×.

Footnotes