Disaggregated Quantization: Splitting the Prefill and Decode Price System Across Two Formats

A September 2026 paper from NVIDIA and ISTA attacks an assumption so baked in that most quantization work never states it: that one model needs one quantized representation¹. Disaggregated Quantization: Specializing LLM Prefill and Decode (Panferov, Kleinegger, Priyadarshi, Blankevoort and Alistarh, arXiv:2609.26333, cs.LG, submitted September 22, 2026) observes that the two phases of LLM inference pay for opposite things. Prefill processes the whole prompt in one shot — that is a GEMM, limited by arithmetic, so low-precision compute formats (NVFP4) accelerate it. Decodes at batch one load every weight from memory to touch one token — that is a memory-traffic problem, so compact weights (1-3 bits) accelerate it. Disaggregated quantization (DQ) therefore specializes computation formats, weights, and even storage placement per phase instead of per model¹.

The paper is not a new quantizer. It is a serving-level observation with a ladder of three increasingly aggressive instantiations: disable activation quantization on decode only (free accuracy); train a separate compute-native NVFP4 prefill checkpoint against the same response objective (faster prefill plus better low-bit accuracy); and stream that second checkpoint from SSD so a single device does not hold both (the 1.78x TTFT claim). The interface between the phases stays the KV cache, whose layer and head dimensions are unchanged, so prefill produces representations that the unchanged decode weights then attend to¹. The catch, which the paper states itself: you now build, store, benchmark, and qualify two artifacts, the SSD trick only pays on long prompts, and the +32.5-point headline is a rescue of a collapsed 1-bit baseline — not parity with BF16.

1. The roofline split: why the phase axes differ

The whole method falls out of one accounting identity. A linear stack with P parameters processing a prompt of L tokens performs about 2·P·L operations. During prefill it loads the P weights once per prompt; during batch-one decode it loads all P weights per generated token to perform just 2·P useful operations. Measured against weight bytes moved, arithmetic intensity is 16·L/bw ops/byte for prefill and 16/bw for decode, where bw is bits per weight — the ratio between the two is exactly the prompt length.

Run it against the paper's hardware, DGX Spark (128 GB unified LPDDR5x at 273 GB/s, 1000 FP4 TOPS with sparsity, i.e. 500e12 dense ops/s on NVIDIA's datasheet accounting)²:

# CELL 1: roofline split -- prefill vs decode vs weight precision
# Arithmetic intensity (AI) measured against WEIGHT bytes, per linear stack.
#   prefill: ops = 2*P*L, weight bytes = P*bw/8 once per prompt -> AI = 16*L/bw
#   decode (batch 1): ops = 2*P, weight bytes = P*bw/8 per token -> AI = 16/bw
# Machine: DGX Spark (NVIDIA spec sheet): 1000 TOPS FP4 *with sparsity*
# -> 500e12 dense ops/s; 273 GB/s LPDDR5x.
 
FP4_DENSE_OPS = 500e12
MEM_BW = 273e9
 
def ai_decode(bw): return 16.0 / bw
def ai_prefill(L, bw): return 16.0 * L / bw
 
bal_fp4 = FP4_DENSE_OPS / MEM_BW
print("CELL 1: AI = ops per weight-byte (ops/byte), DGX Spark")
print(f"machine balance point at dense FP4: {bal_fp4:.0f} ops/byte")
print()
print("weight bits | decode AI | prefill AI L=512 | L=8192 | decode is")
for bw in (16.0, 8.0, 4.5, 2.5, 1.5):
    regime = "compute" if ai_decode(bw) >= bal_fp4 else "bandwidth"
    Lx = bal_fp4 * bw / 16.0
    print(f"  {bw:4.1f}      | {ai_decode(bw):8.2f} | {ai_prefill(512, bw):16.2f} | {ai_prefill(8192, bw):6.0f} | {regime}-bound")
    print(f"             prefill turns compute-bound at L >= {Lx:.0f} tokens")

Output from a real run (Python 3):

CELL 1: AI = ops per weight-byte (ops/byte), DGX Spark
machine balance point at dense FP4: 1832 ops/byte
 
weight bits | decode AI | prefill AI L=512 | L=8192 | decode is
  16.0      |     1.00 |           512.00 |   8192 | bandwidth-bound
             prefill turns compute-bound at L >= 1832 tokens
   8.0      |     2.00 |          1024.00 |  16384 | bandwidth-bound
             prefill turns compute-bound at L >= 916 tokens
   4.5      |     3.56 |          1820.44 |  29127 | bandwidth-bound
             prefill turns compute-bound at L >= 515 tokens
   2.5      |     6.40 |          3276.80 |  52429 | bandwidth-bound
             prefill turns compute-bound at L >= 286 tokens
   1.5      |    10.67 |          5461.33 |  87381 | bandwidth-bound
             prefill turns compute-bound at L >= 172 tokens

Two readings. First, decode sits two to three orders of magnitude below the machine's balance point at every weight precision — compressing weights is the only lever that helps decode, and this stays true however coarse the weights get: at 1.5 bits the decode intensity is still 10.67 ops/byte against a balance point of 1832. Second, prefill crosses into compute-boundedness within a few hundred to a couple thousand tokens — so for real prompts, FP4 arithmetic is what to buy for the prefill side, and nothing about the decode side (where the weights live) fixes prefill speed. This is the whole disagreement with monolithic quantization in one table: a single format can be optimal for at most one phase.

2. The cheapest rung: keep decode weights, drop decode activation quantization

The lowest rung keeps one set of weights and changes only the computation format per phase. NVFP4 quantizes both weights and activations (W4A4), which is what the tensor cores want; its weight-only variant NVFP4A16 leaves activations in higher precision and is more accurate but cannot use the fast FP4 GEMM. The paper's observation: at decode, the act-quant buys nothing (decode is bandwidth-bound — the arithmetic is idle anyway) yet costs accuracy, because the per-token hidden states that carry the response get snapped to the same coarse E2M1 grid prefill already paid for. So: quantize activations during prefill only, skip the activation quantizer during decode. Storage, prefill cost, and decode speed are unchanged or slightly better (skipping act-quant makes decode 2-3% faster), and decode-heavy accuracy improves on all seven tested Qwen 3 and Gemma 3 models¹.

A toy makes the mechanism concrete. Train a small three-layer MLP (standard library only, seed fixed), then walk the weight-bit ladder with block-64 RTQ and toggle a static per-tensor FP4-style activation quantizer on or off:

# CELL 3: accuracy-vs-bits ladder and the decode activation-quantization tax.
# Trained-from-scratch 3-layer MLP (pure stdlib; seed fixed) on an 8-class
# cluster task -- a stand-in for a "released model". Weight quantization is
# block-64 RTQ; activation quantization is per-tensor FP4-style signed grid
# with a STATIC scale set by a running-maximum calibration pass (what NVFP4
# W4A4 does, and what format disaggregation removes on decode).
import random, math
random.seed(2026)
 
D, H, C, N = 64, 96, 8, 1500
centers = [[random.gauss(0, 1) for _ in range(D)] for _ in range(C)]
XN, YN = [], []
for i in range(N):
    c = i % C
    XN.append([random.gauss(centers[c][d], 1.0) for d in range(D)])
    YN.append(c)
 
def init(rows, cols, s): return [[random.gauss(0, s) for _ in range(cols)] for _ in range(rows)]
W1, W2, W3 = init(H, D, (2/D)**0.5), init(H, H, (2/H)**0.5), init(C, H, (2/H)**0.5)
 
def fwd(x_in, Ws, aq=None):
    h = list(x_in)
    for li, W in enumerate(Ws):
        h = [max(0.0, sum(h[k]*W[j][k] for k in range(len(h)))) for j in range(len(W))]
        if aq:
            s = aq[li]
            h = [max(-8*s, min(8*s, round(v/s)*s)) for v in h]  # FP4-ish signed grid
    return h
 
def logits_all(Ws, aq=None): return [fwd(x, Ws, aq) for x in XN]
def acc(lg): return sum(max(range(C), key=lambda j: l[j]) == YN[i] for i, l in enumerate(lg))/N
def softmax_g(l, y):
    m = max(l); es = [math.exp(v-m) for v in l]; S = sum(es)
    g = [es[j]/S for j in range(C)]; g[y] -= 1.0; return g
 
LR = 0.05
for step in range(12):
    for i in range(N):
        hs = [XN[i]]
        for W in (W1, W2, W3):
            hs.append([max(0.0, sum(hs[-1][k]*W[j][k] for k in range(len(hs[-1])))) for j in range(len(W))])
        g3 = softmax_g(hs[3], YN[i])
        g2 = [sum(g3[j]*W3[j][k] for j in range(C))*(1 if hs[2][k] > 0 else 0) for k in range(H)]
        g1 = [sum(g2[j]*W2[j][k] for j in range(H))*(1 if hs[1][k] > 0 else 0) for k in range(H)]
        for j in range(C):
            for k in range(H): W3[j][k] -= LR*g3[j]*hs[2][k]
        for j in range(H):
            for k in range(H): W2[j][k] -= LR*g2[j]*hs[1][k]
        for j in range(H):
            for k in range(D): W1[j][k] -= LR*g1[j]*hs[0][k]
    if step % 30 == 0:
        print(f"  [train] step {step}: acc {acc(logits_all([W1,W2,W3])):.4f}")
 
def rtq_block(W, bits, gs=64):
    Wq = []
    for row in W:
        rq = []
        for i in range(0, len(row), gs):
            blk = row[i:i+gs]
            r = max(abs(w) for w in blk) or 1e-9
            s = 2*r/(2**bits-1)
            rq.extend(round(w/s)*s for w in blk)
        Wq.append(rq)
    return Wq
 
# static per-tensor activation scales via running-max observer on calibration prompts
run_max = [0.0, 0.0, 0.0]
for i in range(200):
    h = XN[i]
    for li, W in enumerate((W1, W2, W3)):
        h = [max(0.0, sum(h[k]*W[j][k] for k in range(len(h)))) for j in range(len(W))]
        run_max[li] = max(run_max[li], max(h))
S = [m/6.0 for m in run_max]   # E2M1 grid saturates ~ +/-6x scale
 
print()
print("CELL 3: accuracy vs weight bits, and the decode act-quant tax")
print("weights | acts | accuracy")
BF = acc(logits_all([W1, W2, W3]))
print(f"BF16    | FP   | {BF:.4f}")
for bits in (4, 3, 2, 1):
    Wq = [rtq_block(W1, bits), rtq_block(W2, bits), rtq_block(W3, bits)]
    a_fp = acc(logits_all(Wq))
    a_aq = acc(logits_all(Wq, aq=S))
    print(f"{bits}-bit   | FP   | {a_fp:.4f}")
    print(f"{bits}-bit   | W4A4 | {a_aq:.4f}    (act-quant tax {a_fp - a_aq:+.4f})")
print(f"BF16    | W4A4 | {acc(logits_all([W1,W2,W3], aq=S)):.4f}    (act-quant alone)")

Output from a real run (Python 3, seed 2026):

  [train] step 0: acc 0.9993
 
CELL 3: accuracy vs weight bits, and the decode act-quant tax
weights | acts | accuracy
BF16    | FP   | 1.0000
4-bit   | FP   | 1.0000
4-bit   | W4A4 | 0.9987    (act-quant tax +0.0013)
3-bit   | FP   | 1.0000
3-bit   | W4A4 | 0.9993    (act-quant tax +0.0007)
2-bit   | FP   | 0.9987
2-bit   | W4A4 | 0.9960    (act-quant tax +0.0027)
1-bit   | FP   | 0.1253
1-bit   | W4A4 | 0.1253    (act-quant tax +0.0000)
BF16    | W4A4 | 0.9987    (act-quant alone)

Read it as the paper's Section 2.2 in miniature. The weight ladder is benign down to 2 bits on this easy toy, then falls off a cliff at 1-bit: naive 1-bit rounding destroys the function outright — the paper's released GGUF decoders survive at 1 bit only because aggressive vector-quantization encodings (IQ1_S and friends) plus training-time recovery do work the toy does not model. And the act-quant tax (the W4A4-vs-FP gap at fixed weight bits) grows as the weights get coarser — the residual headroom shrinks exactly when the weight error is already large, so coarse weights and coarse activations punish each other. That interaction is why removing activation quantization specifically where it is free (bandwidth-bound decode) is the paper's cheapest win: the toy's 2-bit row gives back 0.0027 of accuracy to act-quant that decode never needed to pay. The paper's headline instance of the same effect at scale is its PTQ validation on up-to-2.8T-parameter models, where format disaggregation improves point estimates in 11 of 13 model-benchmark combinations with 6 statistically significant gains and no significant degradations — with nothing retrained¹.

3. Full disaggregation: buy the prefill its own weights

Format disaggregation has a self-imposed ceiling: prefill still computes on a re-quantized view of the 1-3-bit decode weights ("autocast" re-quantizes LUT weights plus activations to NVFP4 on the fly), so the FP4 GEMM runs on inputs already damaged by the compact encoding. The next rung gives prefill its own checkpoint, trained to be compute-native NVFP4 from the start. QADD (quantization-aware distillation with disaggregation) trains both pathways toward one response objective in a single forward-backward pass, using the SFT label mask to route prompt positions through the prefill pathway and response positions through decode; gradients reach the prefill weights through the prompt keys and values that decode later attends to¹. The paper's Table 1 family means (Qwen 3 / Gemma 3, accuracy on decode-heavy and prefill-heavy suites, device GB, speedups over BF16 on DGX Spark) show the shape of the deal for its 2-bit LUT2 rows¹:

• 2-bit weight-only decode: decode-heavy 38.4 / 33.7, 3.82x / 4.15x decode speedup, 4.66 / 5.38 GB.
• LUT2 autocast in both phases (non-disaggregated): decode-heavy collapses to 34.8 / 30.8 — act-quant on decode hurts — at 1.49x / 1.67x prefill speedup.
• Plus format disaggregation: 37.2 / 32.6 — most of the collapse recovered at unchanged device GB.
• Plus full disaggregation: 45.5 / 38.2 decode-heavy and 76.6 / 61.3 prefill-heavy — for LUT3/LUT2 the paper reports gains over the non-disaggregated scheme of 6.3 / 5.2 and 10.7 / 7.4 decode-heavy points respectively — but device memory jumps to 8.57 / 11.43 GB: two checkpoints.
• Plus ODP: the same accuracy at 4.66 / 5.38 GB again, prefill speedup nearly unharmed (1.47x / 1.58x).

So the prefiller is not just a fast prefill; it is a rescue device for low-bit decode. But note what row four bought at listed capacity: memory nearly doubled until ODP gave it back — the subject of the next section — and every fully-disaggregated model now ships as two artifacts.

4. Prefillers for frozen checkpoints: +32.5 points, measured against what

The most commercially pointed experiment keeps the decode side completely untouched: released, pre-quantized Qwen3.8-27B GGUF checkpoints (eight Unsloth releases spanning IQ1_S through Q3_K_XL, in the llama.cpp GGUF ecosystem³), each dequantized and frozen, with only an NVFP4 prefill pathway trained on top¹. The decode quantization pipeline stays a black box — training needs its checkpoint, not its data or algorithm. On the IQ1_S (1-bit) decoder, training the NVFP4 prefiller moves MMLU-Pro from 29.04 to 61.54 (+32.50 points) and MMMU-Pro from 24.39 to 59.65 (+35.26; the abstract rounds it to 35.3), without touching the decode weights — more than doubling 1-bit accuracy on both benchmarks¹. The gains shrink as bitwidth rises — +19.71 MMLU-Pro at IQ1_M, +7.42 at IQ2_XXS, +2.80 at IQ2_S, and turn negative at the 3-bit formats (IQ3_S -0.63, Q3_K_XL -0.62 MMLU-Pro; up to -2.77 MMMU-Pro)¹.

Two facts must travel together. First, the mechanism is representational, not arithmetic: the frozen decoder was already dropping accuracy because its inputs — the prompt KV entries — were being built by a mangled 1-bit prefill comprehension of the prompt; a faithful NVFP4 prompt comprehension restores most of what the decoder can still express. Second, the baseline matters: BF16 Qwen3.8-27B scores 84.62 MMLU-Pro and 75.26 MMMU-Pro in the same table, so the rescued 1-bit pipeline at 61.54 still sits about 23 points below full precision. The +32.5 is the distance from a collapsed 1-bit baseline to a much-less-collapsed one — an argument that 1-bit decode becomes usable when prefill stops sabotaging it, not that it catches BF16¹.

5. ODP: the second checkpoint lives on SSD, and the amortization has a crossover

Full disaggregation on a single device is memory-prohibitive — this is where storage placement becomes part of the quantization decision. Offloaded disaggregated prefill (ODP) exploits a phase-asymmetry: a prefill transformer block's weights are touched exactly once per prompt (as the tokens sweep through layer by layer), then never needed again for the rest of the turn. So the prefill checkpoint can stream from SSD block by block, through two rotating device buffers carved out of the decode weights' own memory (unused during prefill). The device weight footprint returns to the decode-only number; the price is SSD traffic and a fixed loading cost that only long prompts can hide¹.

The amortization math, on the paper's numbers. An NVFP4 checkpoint of a 27B model is about 15.19 GB (27e9 parameters at 4.5 bits/weight — 4 data bits plus the FP8 block scale amortized over 16 elements). That streams once per prompt regardless of length, while prefill compute grows linearly in L:

# CELL 2: ODP SSD amortization for the Qwen3.8-27B NVFP4 prefiller.
# Full disaggregation stores a SECOND checkpoint (~4.5 bpw for NVFP4 incl.
# the FP8 block scales). ODP streams it from SSD: each transformer block is
# needed exactly once per prompt, so SSD time is FIXED while prefill compute
# grows linearly in L. Crossover L*: compute time == SSD load time.
# Calibration: paper reports compute overtakes loading around 8K context on
# DGX Spark; we pin SSD at 3.5 GB/s (ordinary NVMe sustained read) and solve
# for the achieved NVFP4 throughput that makes L* = 8192.
 
P = 27e9          # Qwen3.8-27B parameters
BPW = 4.5         # NVFP4 bits per weight incl. FP8-E4M3 scale per 16 elems
ckpt_gb = P * (BPW / 8) / 1e9
SSD = 3.5         # GB/s sustained
L_star = 8192
 
load_s = ckpt_gb / SSD                       # fixed SSD cost per prompt
T_ops = 2 * P * L_star / load_s             # implied achieved FP4 throughput
print("CELL 2: ODP off the 27B NVFP4 prefiller")
print(f"checkpoint on SSD: {ckpt_gb:.2f} GB at {SSD} GB/s -> fixed load {load_s:.2f} s")
print(f"implied achieved NVFP4 throughput: {T_ops/1e12:.0f} TOPS")
print()
print("L tokens | MB SSD / token | load s | compute s | load/compute")
for L in (1024, 2048, 4096, 8192, 16384, 32768):
    mb = ckpt_gb * 1e3 / L
    cs = 2 * P * L / T_ops
    print(f"{L:8d} | {mb:15.2f} | {load_s:6.2f} | {cs:9.2f} | {load_s/cs:11.2f}")
print()
print("crossover sensitivity (L* scales with SSD bandwidth):")
for g in (1.75, 3.5, 7.0):
    print(f"  SSD {g:4.2f} GB/s -> L* = {L_star * SSD / g:6.0f} tokens")

Output from a real run (Python 3):

CELL 2: ODP off the 27B NVFP4 prefiller
checkpoint on SSD: 15.19 GB at 3.5 GB/s -> fixed load 4.34 s
implied achieved NVFP4 throughput: 102 TOPS
 
L tokens | MB SSD / token | load s | compute s | load/compute
    1024 |           14.83 |   4.34 |      0.54 |        8.00
    2048 |            7.42 |   4.34 |      1.08 |        4.00
    4096 |            3.71 |   4.34 |      2.17 |        2.00
    8192 |            1.85 |   4.34 |      4.34 |        1.00
   16384 |            0.93 |   4.34 |      8.68 |        0.50
   32768 |            0.46 |   4.34 |     17.36 |        0.25
 
crossover sensitivity (L* scales with SSD bandwidth):
  SSD 1.75 GB/s -> L* =  16384 tokens
  SSD 3.50 GB/s -> L* =   8192 tokens
  SSD 7.00 GB/s -> L* =   4096 tokens

The structural facts the table exposes. Loading dominates by 8-to-1 at 1K tokens and is still 2x compute at 4K — which is exactly why the paper reports ODP being slower than the resident weight-only baseline at short prompts and only pulling ahead from about 4K context. The crossover lands at 8K on the paper's DGX Spark measurements (TTFT 1.78x faster than the weight-only IQ1_S baseline at 8K in their llama.cpp fork), and the implied achieved FP4 throughput there — 102 TOPS, a fifth of the dense peak — is a healthy sanity anchor for the calibration. And the crossover is a property of the ratio of SSD bandwidth to achieved compute, not of the model size: halve your SSD and your crossover doubles. None of this helps a MoE model, as the paper notes — active-parameter compute per loaded byte is low, so loading stays expensive up to extreme context lengths¹.

6. Anti-hype: the invoice for phase-splitting

The paper's own limitations section and tables price the method honestly; the invoice has five lines.

Two artifacts to store, qualify, and keep consistent. A fully-disaggregated deployment owns a decode checkpoint and a prefill checkpoint (Qwen 3 family Table 1: 8.57 GB total for 2-bit vs 4.66 GB decode-only). ODP removes the device residency, not the artifact: 15.19 GB sits on SSD and must be versioned, downloaded, and regression-tested against its decoder — and each decode bitwidth is paired with its own prefiller in the paper's setup, so the pairing matrix grows.

The ODP win is prompt-length-conditional. Below roughly 4K tokens the SSD load — 4.34 s of fixed streaming traffic for the 27B prefiller at the calibration above — is pure added TTFT latency. Any deployment whose traffic is chat-length bursts gets the memory savings and the slowdown; the 1.78x is an 8K-context number, reached from a deficit below 4K¹.

The prefiller is trained, not downloaded — except when it is. The +32.5-point prefiller consumed a QADD run on reasoning traces distilled from the BF16 model (opting out of full disaggregation is not free). The released artifacts cover Qwen3.8-27B; every other decoder gets the bill for its own training run¹.

The 1-3-bit decode tax survives the rescue. Rescued IQ1_S scores 61.54 MMLU-Pro against 84.62 for BF16 in the same appendix table — the prefiller recovers the damage the 1-bit prefill view was doing, not the damage the 1-bit weights do. And above 2 bits the prefiller can subtract: the 3-bit formats lose up to 2.77 MMMU-Pro points with the NVFP4 prefiller attached. There is no bitwidth at which the second checkpoint is unconditionally positive¹.

The split presupposes disaggregated plumbing. The accuracy numbers ride on real disaggregated serving (vLLM plus NIXL, KV-cache transfer between engines), and the ODP numbers on a custom llama.cpp fork. All of it is measured at batch one, single-turn: the paper does not evaluate high-concurrency batching, and it flags multi-turn chat as untested — re-prefilling cached assistant tokens through the prefill checkpoint can yield different KV entries than the tokens' decode-time representations, and nobody has measured how much that matters¹.

7. Verdict

Disaggregated quantization is a serving-level reframing with an unusually clean mechanism: the roofline makes the two phases want opposite things, and the paper stops forcing one format to serve both. The cheap rung — skip activation quantization on decode — is close to unconditionally good, and the paper's PTQ validation on production-scale models up to 2.8T parameters (11 of 13 model-benchmark combinations improved, none significantly degraded) supports that rung being adopted as a default in disaggregated serving. The expensive rungs are a genuine engineering trade: a second, trained checkpoint exchanged for large low-bit accuracy recoveries and FP4 prefill speed, with the SSD streaming turning the memory cost into a latency cost that only long prompts amortize. The honest summary of the 1-bit result is that phase-splitting moves a 1-bit pipeline from broken to usable — and the distance that remains to BF16 is the standing research problem, not a solved one.

Footnotes