Disaggregated Quantization: Splitting the Prefill and Decode Price System Across Two Formats

A September 2026 paper from NVIDIA and ISTA attacks an assumption so baked in that most quantization work never states it: that one model needs one quantized representation¹. Disaggregated Quantization: Specializing LLM Prefill and Decode (Panferov, Kleinegger, Priyadarshi, Blankevoort and Alistarh, arXiv:2609.26333, cs.LG, submitted September 22, 2026) observes that the two phases of LLM inference pay for opposite things. Prefill processes the whole prompt in one shot — that is a GEMM, limited by arithmetic, so low-precision compute formats (NVFP4) accelerate it. Decodes at batch one load every weight from memory to touch one token — that is a memory-traffic problem, so compact weights (1-3 bits) accelerate it. Disaggregated quantization (DQ) therefore specializes computation formats, weights, and even storage placement per phase instead of per model¹.

The paper is not a new quantizer. It is a phase-specific serving and training approach with three increasingly involved instantiations: disable activation quantization on decode only; train a separate compute-native NVFP4 prefill checkpoint against the same response objective; and stream that second checkpoint from SSD. The interface between phases remains the KV cache, whose layer and head dimensions are unchanged¹. The separate-checkpoint schemes require two artifacts to store and qualify. The 1.78x TTFT is an 8K-context result in a custom llama.cpp fork, and the +32.5-point accuracy result does not reach BF16 parity¹.

1. The roofline split: why the phase axes differ

The whole method falls out of one accounting identity. A linear stack with P parameters processing a prompt of L tokens performs about 2·P·L operations. During prefill it loads the P weights once per prompt; during batch-one decode it loads all P weights per generated token to perform just 2·P useful operations. Measured against weight bytes moved, arithmetic intensity is 16·L/bw ops/byte for prefill and 16/bw for decode, where bw is bits per weight — the ratio between the two is exactly the prompt length.

For an illustrative balance calculation, use DGX Spark's 273 GB/s memory bandwidth and half of its advertised 1 PFLOP sparse FP4 peak as a nominal dense FP4 peak. NVIDIA publishes the sparse theoretical figure, not a measured 500 TOPS dense throughput. Applying this FP4 reference to BF16 or custom LUT rows does not predict their actual kernel crossover².

# CELL 1: roofline split -- prefill vs decode vs weight precision
# Arithmetic intensity (AI) measured against WEIGHT bytes, per linear stack.
#   prefill: ops = 2*P*L, weight bytes = P*bw/8 once per prompt -> AI = 16*L/bw
#   decode (batch 1): ops = 2*P, weight bytes = P*bw/8 per token -> AI = 16/bw
# Machine: DGX Spark (NVIDIA spec sheet): 1000 TOPS FP4 *with sparsity*
# -> nominal 500e12 dense ops/s; 273 GB/s LPDDR5x.
 
FP4_DENSE_OPS = 500e12
MEM_BW = 273e9
 
def ai_decode(bw): return 16.0 / bw
def ai_prefill(L, bw): return 16.0 * L / bw
 
bal_fp4 = FP4_DENSE_OPS / MEM_BW
print("CELL 1: AI = ops per weight-byte (ops/byte), DGX Spark")
print(f"machine balance point at dense FP4: {bal_fp4:.0f} ops/byte")
print()
print("weight bits | decode AI | prefill AI L=512 | L=8192 | decode is")
for bw in (16.0, 8.0, 4.5, 2.5, 1.5):
    regime = "compute" if ai_decode(bw) >= bal_fp4 else "bandwidth"
    Lx = bal_fp4 * bw / 16.0
    print(f"  {bw:4.1f}      | {ai_decode(bw):8.2f} | {ai_prefill(512, bw):16.2f} | {ai_prefill(8192, bw):6.0f} | {regime}-bound")
    print(f"             nominal FP4-reference crossover L >= {Lx:.0f} tokens")

Output from a real run (Python 3):

CELL 1: AI = ops per weight-byte (ops/byte), DGX Spark
machine balance point at dense FP4: 1832 ops/byte
 
weight bits | decode AI | prefill AI L=512 | L=8192 | decode is
  16.0      |     1.00 |           512.00 |   8192 | bandwidth-bound
             nominal FP4-reference crossover L >= 1832 tokens
   8.0      |     2.00 |          1024.00 |  16384 | bandwidth-bound
             nominal FP4-reference crossover L >= 916 tokens
   4.5      |     3.56 |          1820.44 |  29127 | bandwidth-bound
             nominal FP4-reference crossover L >= 515 tokens
   2.5      |     6.40 |          3276.80 |  52429 | bandwidth-bound
             nominal FP4-reference crossover L >= 286 tokens
   1.5      |    10.67 |          5461.33 |  87381 | bandwidth-bound
             nominal FP4-reference crossover L >= 172 tokens

The weight-only arithmetic intensity rises with prompt length and is low for batch-one decode. The printed crossover lengths are toy estimates under the same FP4 peak for every row; BF16 and LUT kernels have different compute ceilings, and attention, activation traffic, scales and other operations are omitted. The paper's measured batch-one results, rather than this simplified balance line, support the conclusion that compact weights help decode while native FP4 computation helps long prefill¹.

2. The cheapest rung: keep decode weights, drop decode activation quantization

The lowest rung keeps one set of weights and changes only the computation format per phase. NVFP4 quantizes both weights and activations (W4A4), which is what the tensor cores want; its weight-only variant NVFP4A16 leaves activations in higher precision and is more accurate but cannot use the fast FP4 GEMM. The paper's observation: at decode, the act-quant buys nothing (decode is bandwidth-bound — the arithmetic is idle anyway) yet costs accuracy, because the per-token hidden states that carry the response get snapped to the same coarse E2M1 grid prefill already paid for. So: quantize activations during prefill only, skip the activation quantizer during decode. Storage, prefill cost, and decode speed are unchanged or slightly better (skipping act-quant makes decode 2-3% faster), and decode-heavy accuracy improves on all seven tested Qwen 3 and Gemma 3 models¹.

The paper's own Table 1 is a better check of the activation-quantization effect than a synthetic MLP. Its family means on decode-heavy tasks compare the same stored weights with activation quantization on both phases ("uniform") versus off during decode ("format disaggregated")¹:

# CELL 3: percentage-point changes from the paper's Table 1 family means.
rows = {
    "Qwen 3, NVFP4": (61.8, 63.7),
    "Gemma 3, NVFP4": (51.9, 55.0),
    "Qwen 3, LUT2": (34.8, 37.2),
    "Gemma 3, LUT2": (30.8, 32.6),
}
for name, (uniform, split) in rows.items():
    print(f"{name}: {split - uniform:+.1f} percentage points")

Qwen 3, NVFP4: +1.9 percentage points
Gemma 3, NVFP4: +3.1 percentage points
Qwen 3, LUT2: +2.4 percentage points
Gemma 3, LUT2: +1.8 percentage points

These are family averages for the paper's QADD experiments, not a guarantee for every workload. Its separate post-training-quantization evaluation improves 11 of 13 model–benchmark point estimates, with six significant gains and no significant degradations at its per-comparison threshold. One of those eight models, Kimi-K3, uses MXFP4 rather than NVFP4; Qwen3.8-2.4T's unquantized reference is FP8¹.

3. Full disaggregation: buy the prefill its own weights

Format disaggregation has a self-imposed ceiling: prefill still computes on a re-quantized view of the 1-3-bit decode weights ("autocast" re-quantizes LUT weights plus activations to NVFP4 on the fly), so the FP4 GEMM runs on inputs already damaged by the compact encoding. The next rung gives prefill its own checkpoint, trained to be compute-native NVFP4 from the start. QADD (quantization-aware distillation with disaggregation) trains both pathways toward one response objective in a single forward-backward pass, using the SFT label mask to route prompt positions through the prefill pathway and response positions through decode; gradients reach the prefill weights through the prompt keys and values that decode later attends to¹. Table 1 averages accuracy across Qwen 3 and Gemma 3 model sizes; its device-memory and speed columns instead use Qwen3-8B and Gemma3-12B on DGX Spark. Its 2-bit LUT2 rows show the trade-off¹:

• 2-bit weight-only decode: decode-heavy 38.4 / 33.7, 3.82x / 4.15x decode speedup, 4.66 / 5.38 GB.
• LUT2 autocast in both phases (non-disaggregated): decode-heavy collapses to 34.8 / 30.8 — act-quant on decode hurts — at 1.49x / 1.67x prefill speedup.
• Plus format disaggregation: 37.2 / 32.6 — most of the collapse recovered at unchanged device GB.
• Plus full disaggregation: 45.5 / 38.2 decode-heavy and 76.6 / 61.3 prefill-heavy — for LUT3/LUT2 the paper reports gains over the non-disaggregated scheme of 6.3 / 5.2 and 10.7 / 7.4 decode-heavy points respectively — but device memory jumps to 8.57 / 11.43 GB: two checkpoints.
• Plus ODP: the same accuracy at 4.66 / 5.38 GB again, prefill speedup nearly unharmed (1.47x / 1.58x).

So the prefiller is not just a fast prefill; it is a rescue device for low-bit decode. But note what row four bought at listed capacity: memory nearly doubled until ODP gave it back — the subject of the next section — and every fully-disaggregated model now ships as two artifacts.

4. Prefillers for frozen checkpoints: +32.5 points, measured against what

The frozen-decoder experiment starts from eight Unsloth Qwen3.8-27B GGUF checkpoints, IQ1_S through Q3_K_XL³. For training and accuracy evaluation, the authors dequantize each GGUF decoder to BF16 and freeze its numerical weights; they train only the NVFP4 prefill pathway. The MMLU-Pro/MMMU-Pro scores therefore do not measure low-bit decode-kernel speed or resident GGUF memory. The separate llama.cpp experiment measures TTFT with the compressed IQ1_S checkpoint¹. The decode quantization pipeline stays a black box: prefiller training needs its checkpoint, not its quantization data or algorithm. On the IQ1_S (1-bit) decoder, training the NVFP4 prefiller moves MMLU-Pro from 29.04 to 61.54 (+32.50 points) and MMMU-Pro from 24.39 to 59.65 (+35.26; the abstract rounds it to 35.3), without touching the decode weights — more than doubling 1-bit accuracy on both benchmarks¹. The gains generally shrink at higher bitwidths — +19.71 MMLU-Pro at IQ1_M, +7.42 at IQ2_XXS and +2.80 at IQ2_S — but are not monotonic. IQ3_XXS still gains +0.72 MMLU-Pro; IQ3_S and Q3_K_XL lose -0.63 and -0.62 MMLU-Pro, and the tested 3-bit formats lose up to 2.77 MMMU-Pro points¹.

Two facts must travel together. First, the results are consistent with a representational mechanism: the prefiller changes prompt-derived KV entries while the decoder weights remain fixed. The paper does not isolate a unique causal contribution of each changed representation. Second, the baseline matters: BF16 Qwen3.8-27B scores 84.62 MMLU-Pro and 75.26 MMMU-Pro in the same table, so the rescued 1-bit pipeline at 61.54 still sits about 23 points below full precision. The +32.5 is the distance from a collapsed 1-bit baseline to a much-less-collapsed one — evidence of a large accuracy recovery under this evaluation protocol, not BF16 parity¹.

5. ODP: the second checkpoint lives on SSD, and the amortization has a crossover

Full disaggregation on a single device is memory-prohibitive — this is where storage placement becomes part of the quantization decision. Offloaded disaggregated prefill (ODP) exploits a phase-asymmetry: a prefill transformer block's weights are touched exactly once per prompt (as the tokens sweep through layer by layer), then never needed again for the rest of the turn. So the prefill checkpoint can stream from SSD block by block, through two rotating device buffers carved out of the decode weights' own memory (unused during prefill). The device weight footprint returns to the decode-only number; the price is SSD traffic and a fixed loading cost that only long prompts can hide¹.

The following illustrative model combines the paper's nominal 27B parameter count and 4.5 bits per NVFP4 weight with an assumed 3.5 GB/s sustained SSD rate. Its 15.19 GB is an approximate computed weight payload, not a measured checkpoint size; other tensors, metadata and extra carve-out traffic are omitted. Streaming cost is approximately fixed per prompt, while the linear-layer compute term grows with prompt length:

# CELL 2: ODP SSD amortization for the Qwen3.8-27B NVFP4 prefiller.
# Full disaggregation stores a SECOND checkpoint (~4.5 bpw for NVFP4 incl.
# the FP8 block scales). ODP streams it from SSD: each transformer block is
# needed exactly once per prompt, so SSD time is FIXED while prefill compute
# grows linearly in L. Crossover L*: compute time == SSD load time.
# Illustration: paper reports compute overtaking SSD load around 8K for
# its Qwen 3 models on DGX Spark. Assume 3.5 GB/s for this 27B toy and
# back-solve throughput that makes its modeled crossover L* = 8192.
 
P = 27e9          # Qwen3.8-27B parameters
BPW = 4.5         # NVFP4 bits per weight incl. FP8-E4M3 scale per 16 elems
ckpt_gb = P * (BPW / 8) / 1e9
SSD = 3.5         # GB/s sustained
L_star = 8192
 
load_s = ckpt_gb / SSD                       # fixed SSD cost per prompt
T_ops = 2 * P * L_star / load_s             # back-solved FP4 throughput, not a measurement
print("CELL 2: ODP off the 27B NVFP4 prefiller")
print(f"estimated weight payload: {ckpt_gb:.2f} GB at {SSD} GB/s -> fixed load {load_s:.2f} s")
print(f"back-solved NVFP4 throughput: {T_ops/1e12:.0f} TOPS")
print()
print("L tokens | MB SSD / token | load s | compute s | load/compute")
for L in (1024, 2048, 4096, 8192, 16384, 32768):
    mb = ckpt_gb * 1e3 / L
    cs = 2 * P * L / T_ops
    print(f"{L:8d} | {mb:15.2f} | {load_s:6.2f} | {cs:9.2f} | {load_s/cs:11.2f}")
print()
print("crossover sensitivity (L* scales with SSD bandwidth):")
for g in (1.75, 3.5, 7.0):
    print(f"  SSD {g:4.2f} GB/s -> L* = {L_star * SSD / g:6.0f} tokens")

Output from a real run (Python 3):

CELL 2: ODP off the 27B NVFP4 prefiller
estimated weight payload: 15.19 GB at 3.5 GB/s -> fixed load 4.34 s
back-solved NVFP4 throughput: 102 TOPS
 
L tokens | MB SSD / token | load s | compute s | load/compute
    1024 |           14.83 |   4.34 |      0.54 |        8.00
    2048 |            7.42 |   4.34 |      1.08 |        4.00
    4096 |            3.71 |   4.34 |      2.17 |        2.00
    8192 |            1.85 |   4.34 |      4.34 |        1.00
   16384 |            0.93 |   4.34 |      8.68 |        0.50
   32768 |            0.46 |   4.34 |     17.36 |        0.25
 
crossover sensitivity (L* scales with SSD bandwidth):
  SSD 1.75 GB/s -> L* =  16384 tokens
  SSD 3.50 GB/s -> L* =   8192 tokens
  SSD 7.00 GB/s -> L* =   4096 tokens

In this assumed model, load time exceeds linear-layer compute time below 8K. The 102 TOPS output is back-solved from the chosen 8K crossover and assumed SSD speed, not an observed throughput. The paper separately measures compute overtaking SSD load around 8K in its Qwen 3 DGX Spark experiments; for Qwen3.8-27B in llama.cpp, ODP becomes faster than weight-only IQ1_S at 4K and gives 12.27 s versus 6.90 s TTFT at 8K. These are different comparisons, so the toy crossover does not predict the 4K baseline crossing. A slower SSD raises the modeled load/compute crossover, conditional on unchanged compute rate. The authors find ODP less useful for MoE models because active compute per loaded parameter is lower¹.

6. Anti-hype: the invoice for phase-splitting

The paper's own limitations section and tables price the method honestly; the invoice has five lines.

Two artifacts to store, qualify, and keep consistent. A fully-disaggregated deployment owns a decode checkpoint and a prefill checkpoint (Table 1 for Qwen3-8B: 8.57 GB device memory for full 2-bit disaggregation vs 4.66 GB for weight-only). ODP removes the device residency, not the artifact: an additional prefiller checkpoint sits on SSD (the 15.19 GB figure above is only a payload estimate) and must be versioned, downloaded, and regression-tested against its decoder — and each decode bitwidth is paired with its own prefiller in the paper's setup, so the pairing matrix grows.

The ODP win is prompt-length-conditional. In the paper's llama.cpp run ODP is slower than the weight-only baseline below 4K tokens; the toy's 4.34 s SSD transfer estimate is not a measured TTFT component. Any deployment whose traffic is chat-length bursts gets the memory savings and the slowdown; the 1.78x is an 8K-context number, reached from a deficit below 4K¹.

The prefiller is trained, not downloaded — except when it is. The +32.5-point prefiller consumed a QADD run on reasoning traces distilled from the BF16 model (opting out of full disaggregation is not free). The released artifacts cover Qwen3.8-27B; every other decoder gets the bill for its own training run¹.

The low-bit accuracy and generation-cost questions survive the rescue. Rescued IQ1_S scores 61.54 MMLU-Pro against 84.62 for BF16 in the same appendix table — the prefiller recovers the damage the 1-bit prefill view was doing, not the damage the 1-bit weights do. And above 2 bits the prefiller can subtract: the 3-bit formats lose up to 2.77 MMMU-Pro points with the NVFP4 prefiller attached. There is no bitwidth at which the second checkpoint is unconditionally positive. Nor does a fixed decoder guarantee fixed total generation cost: in Appendix B.5 the prefiller changes response lengths and truncation rates, sometimes substantially¹.

The implementations differ. The accuracy numbers use disaggregated vLLM plus NIXL with dequantized GGUF decoders, whereas the ODP TTFT numbers use a custom llama.cpp fork and the compressed IQ1_S baseline. All of it is measured at batch one, single-turn: the paper does not evaluate high-concurrency batching, and it flags multi-turn chat as untested — re-prefilling cached assistant tokens through the prefill checkpoint can yield different KV entries than the tokens' decode-time representations, and nobody has measured how much that matters¹.

7. Verdict

Disaggregated quantization is a serving-level reframing with an unusually clean mechanism: the roofline makes the two phases want opposite things, and the paper stops forcing one format to serve both. The cheap rung — skip activation quantization on decode — improves 11 of 13 point estimates in the paper's large-model PTQ evaluation, with six significant gains and no significant degradations under its per-comparison tests. That supports testing the variant for a given model and workload; it does not establish a universal default. The expensive rungs are a genuine engineering trade: a second, trained checkpoint exchanged for large low-bit accuracy recoveries and FP4 prefill speed, with the SSD streaming turning the memory cost into a latency cost that only long prompts amortize. The honest summary of the 1-bit result is that phase-splitting substantially raises this IQ1_S decoder's measured benchmark accuracy — and the distance that remains to BF16 is the standing research problem, not a solved one.

Footnotes