Quantization Can Change Retrieval Results While Classification Accuracy Holds

A September 2026 paper documents a failure mode that classification accuracy alone cannot detect: a 4-bit quantized model can change a large fraction of its retrieval top-1 results while changing far fewer classification predictions. The Undetected Damage of Quantization on Retrieval and How to Fix It (Zhou, Zirilli, Solombrino, Dessì and Rodolà, arXiv:2609.24322, submitted September 21, 2026) reports top-1 retrieval changes for roughly 14 to 46 percent of queries across its tested models and settings. Its ViT comparisons use the same fine-tuned checkpoint for both tasks; one Qwen3 comparison also controls for the checkpoint, while the first Qwen3 row compares different checkpoints¹. On the paper's CLIP text-to-image setting, Recall@1 falls by 1.3 percent while 24.6 percent of top-1 results change and 10.9 percent of previously correct queries lose the correct image¹.

The mechanism is a score-gap bound. If quantization changes every score — a class logit or a query-document similarity — by at most ε, no other candidate can strictly overtake the top item when its lead over the runner-up is at least 2ε; a smaller lead admits a perturbation that swaps them. The paper finds wider top gaps for its classification tasks than for its retrieval tasks. A contrastive loss against sampled negatives does not generally guarantee separation from every candidate in the evaluated corpus¹. The gap is measurable without relevance labels, but the bound's ε or a deployable threshold still has to be measured or calibrated. The paper proposes extra bit-width for gap-sensitive layers and routing low-gap inputs to full precision.

This guide verifies the numbers at primary, walks the bound and the assumptions it quietly makes, reproduces the failure mode, the bound check and the selective-precision economics in runnable Python, and then prices the anti-hype: what a "top-1 change" is and is not, and where the clean bound stops being clean.

1. The effect: same model, same weights, different task, different damage

The paper's core experiment holds everything fixed except the task. Three fine-tuned backbones — ViT-B/16 and ViT-L/16 for images, Qwen3-Embedding-0.6B for text, 53 fine-tuned checkpoints in total — are quantized weight-only at 4 and 3 bits (W4/W3), symmetric per-channel or per-128-weight-group scales, every linear layer quantized, and then read two ways: as a classifier (top-1 predicted class) and as a retriever (first-ranked item over a corpus)¹. At W4 round-to-nearest with group-128 scales, the top-1 change rate versus the full-precision (FP) model splits by task¹:

The same checkpoint, the same quantizer, the same bit-width: a few percent of classification answers change, and up to 45.5 percent of retrieval top-1 results do. The abstract's headline range of 14 to 46 percent spans the paper's full measurement set — 14 percent is the best case (a well-separated text embedder such as GTE or BGE under a stronger quantizer like GPTQ at W4), and the upper end is image retrieval under plain round-to-nearest¹. The single-checkpoint row of the paper's Table 1 makes the asymmetry un-ignorable for the "different models" objection: one set of embeddings, read through a frozen linear probe versus cosine self-retrieval, still changes retrieval top-1 for 41.2 percent of queries versus 8.6 percent for classification, a 4.8× ratio on identical vectors¹.

Aggregate metrics can understate the per-query changes. In the paper's Table 2 (round-to-nearest, group-128, four BEIR corpora), Qwen3-Emb-8B at W4 loses 3.1 percent of mean nDCG@10 while 15.6 percent of queries whose relevant document ranked first in FP lose it from rank 1. Qwen3-Emb-0.6B loses 7.5 percent of mean nDCG@10 while 22.2 percent of such queries lose that document from rank 1. The table separately reports loss from the top 10 among queries with a relevant document originally in the FP top 10; those percentages have a different denominator and range from 15.3 to 23.7 percent for the four W4 rows. Changes to a ranked list and changes to mean relevance metrics answer different questions¹.

2. The mechanism: gap, contender set, and the 2ε bound

The analysis is a worst-case argument over the score vector z over N candidates, where a "score" is whatever the system ranks by — a class logit in classification, a query-candidate similarity in retrieval. Quantization is modeled as replacing z with z + δ, where δ is a perturbation with ‖δ‖∞ ≤ ε, the largest amount by which quantization displaces any score. Writing g_2(z) for the gap between the top-1 and the runner-up, the paper's Lemma 1 states: the top-1 survives every ε-bounded perturbation if and only if g_2(z) ≥ 2ε¹.

Both directions matter. Necessity is the two-line argument: each score moves by at most ε, so a candidate can overtake the winner only if it started within 2ε of it; when nobody did, no perturbation can produce a new leader. Sufficiency is the constructed swap: if the gap is below 2ε, subtract ε from the winner and add ε to the runner-up and the order inverts. The proof sits on older ground than the paper — when every score is known only to an interval of the same width, the induced comparison relation is the semiorder studied by Luce, Scott and Suppes, and Fishburn, and the rankings a bounded perturbation can reach are its linear extensions². The paper's own credit is precise: "what is new here is the measurement, not the condition"¹.

Three refinements make the condition deployable rather than merely true.

The contender set, not just the runner-up. The set C_ε(z) = {k ≥ 2 : g_k(z) < 2ε} contains positions whose candidates can strictly overtake the FP top-1 under an ε-bounded perturbation. It can extend far down the ranked list when many scores cluster within 2ε of the winner. A chain of pairwise near-ties cannot enable a candidate whose own gap from the winner is at least 2ε to overtake it: this follows directly from the score bound. The preprint's sentence attributing a far-below-winner overtake to nontransitivity is therefore incorrect if “far below” means a score gap of at least 2ε¹. In its experiments, the new top-1 was the old runner-up in 82.2 percent of classification flips but only 39.7 percent of image-retrieval flips; the other winners were lower ranked yet still within the allowed score gap¹.

Differences, not levels. Corollary 1 observes that whether the top-1 survives depends on δ only through the differences δ_j − δ_a involving the winner — a perturbation that shifts every score equally changes nothing, however big ε is. So the quantity to calibrate is S(x) = max_j(δ_a′ − δ_j) ≤ 2‖δ‖∞ on the realized perturbation, not the global 2ε; the paper's stability check therefore accepts strictly more inputs than the raw bound would (calibrating 2ε instead of S accepts 67.2 percent of classification inputs against 85.2 percent at α = 0.10, and no retrieval query at all)¹.

Measured, not assumed. Crucially, the paper does not assume ε: it runs both the FP model and the quantized model on the input and measures δ = z^PTQ − z^FP, taking ε = ‖δ‖∞. The bound's job is structural — it tells you which statistic predicts flips — while the perturbation size is an empirical quantity, which is what lets one threshold generalize across queries.

The measured gaps help explain the task asymmetry in the tested settings. At W4 the contender set is empty for 74.3 percent of ViT-B/16 classification inputs and for none of the retrieval queries; over the paper's measured inputs, 85.8 percent of classification inputs exceed the stability threshold (median separation ratio g_2/2ε = 5.73) versus 3.5 percent of retrieval queries (median 0.081)¹. Classification training encourages a gap against other classes. Contrastive retrieval training against sampled negatives does not guarantee that its highest-scoring result will be separated from the runner-up in the full evaluation corpus.

3. What the gap buys: prediction, per-input trust, and two fixes

The separation ratio g_2/2ε turns out to explain the damage across everything the paper measures: the median separation correlates with the top-1 change rate at Spearman −0.88 over 2,118 measured configurations, and −0.98 across twelve model/bit-width configurations (p < 10⁻⁷)¹. Neither ingredient alone is as predictive — the median gap reaches −0.58 and the median ε just +0.05, against −0.88 for the ratio. A practitioner's consequence the paper makes explicit: pick the retrieval model with higher separation, not the larger one — GTE-large at 0.34B parameters changes fewer top-1 results at both bit-widths than Qwen3-Embedding-4B at twelve times the size, because its gaps are wider¹.

Per-input stability check (no labels). The top-1 condition needs FP scores and the perturbation size, neither available when only the quantized model runs. The paper instead compares the quantized model's top-1/top-2 gap with a threshold calibrated from unlabeled inputs on which both models are run. For an exchangeable calibration set of size n, the valid upper-tail threshold for S(x) is the k-th smallest calibration value, k = ⌈(n+1)(1−α)⌉ (or +∞ if k > n). It bounds the joint probability Pr(accepted and top-1 changed) by α; it does not guarantee that the error rate conditional on acceptance is at most α³. The preprint says “k-th largest” at this point; that rank direction is inconsistent with the stated upper-tail guarantee. Its reported Table 3 values are empirical: on ViT-B/16 classification at α = 0.10, coverage is 85.2 ± 0.1 percent and the conditional violation rate is 1.47 percent; on text retrieval coverage is 11.3 percent, and on CLIP ViT-L/14 it is 15.7 percent¹. The paper's results do not independently establish which order statistic the implementation used. Any conformal guarantee also needs exchangeability between calibration and deployment inputs and can fail under distribution shift.

Fix 1, retrieval: allocate bits where the gap moves. Weight-only PTQ methods decide how to round, not where to spend bits — bit-width is set globally and existing allocators rank layers by reconstruction error. The paper's criterion instead spends the extra bit-width on the layers whose quantization moves the top-1/top-2 gap most: for each linear layer ℓ, quantize that layer alone with plain round-to-nearest at 3 bits, rerun 128 calibration queries, and take s_ℓ = median_x |g_2(x) − g_2^(ℓ)(x)| — how much the one-layer quantization displaces the gap between the same two documents. Layers are ranked by s_ℓ divided by parameter count, and at a fixed budget of 3.5 bits per weight, the top half receives 4 bits and the rest stay at 3. The whole measurement is L+1 forward passes over 128 unlabeled queries and trains nothing; the sensitivities are measured once with RTN and the resulting assignment is applied under whatever quantizer the deployment uses¹. The result: gap allocation recovers 60-73 percent of the full W3→W4 benefit at half the extra bit's cost, ranks first among all seven forward-only criteria under every quantizer tested, and its lead over the next-best criterion runs from 5.5 to 18.4 points across 24 model/corpus/quantizer configurations. The allocator it replaces — reconstruction error, the objective of existing allocation methods — captures only 40-48 percent and is nearly uncorrelated with gap sensitivity across layers on text embedders (mean Spearman −0.04)¹. In absolute terms at 3.5 bits under RTN: 33.3 percent of top-1 results change against 43.1 percent for reconstruction-driven allocation, and 13.8 percent of correct top-1 documents are lost against 19.9 percent¹.

Fix 2, classification: route low-gap inputs to FP. In classification the at-risk minority is small, so the fix is per-input. The quantized model's own top-1/top-2 gap comes free from the forward pass the deployment already runs; route x to the FP model iff g_q(x) < τ, with τ set at the 25th percentile of g_q on an unlabeled validation slice¹. Routing 25 percent of inputs recovers 85-93 percent of the accuracy lost to quantization on the paper's three backbones (tasks with at least a 0.5-point FP→PTQ gap), at a cost the paper computes as 47 percent of the FP cost at the 4.5× speedup GPTQ reports. A Proposition 3 identifiability result adds a theoretical floor: up to order-equivalence, the gap is the only score of the quantized logits whose threshold sets match the worst-case flippable sets at every budget¹.

4. Runnable cells

Three self-contained cells (standard library only, seed fixed, run with Python 3.9.6). The first reproduces the asymmetry: identical bounded perturbations, two different gap distributions, and the flip-rate gap that results.

import random
 
# CELL 1: the gap>2*eps criterion -- retrieval vs classification gap distributions
random.seed(11)
 
N = 200            # candidates
Q = 500            # queries/inputs per task
 
def retrieval_scores():
    # retrieval-like: scores cluster near the top (nothing separates 1st from 2nd)
    return sorted((random.uniform(0.85, 1.0) for _ in range(N)), reverse=True)
 
def classification_scores():
    # classification-like: winner pulled away from the pack by a wide margin
    rest = sorted((random.uniform(0.0, 0.35) for _ in range(N - 1)), reverse=True)
    return [0.9 + random.uniform(0.0, 0.1)] + rest
 
eps = 0.01          # largest rounding-induced score displacement
 
def flip_stats(gen):
    flips = contenders = 0
    for _ in range(Q):
        z = gen()
        g2 = z[0] - z[1]
        if g2 < 2 * eps:
            contenders += 1
        # simulate the actual perturbation: independent uniform in [-eps, eps]
        d = [random.uniform(-eps, eps) for _ in range(N)]
        zp = [a + b for a, b in zip(z, d)]
        if zp.index(max(zp)) != 0:
            flips += 1
    return flips / Q, contenders / Q
 
rf, rc = flip_stats(retrieval_scores)
cf, cc = flip_stats(classification_scores)
 
print(f"eps = {eps}  (stability threshold 2*eps = {2*eps})")
print()
print("task          | top-1 flip rate | at-risk inputs (g2 < 2*eps)")
print(f"retrieval     |     {rf:6.1%}      |     {rc:6.1%}")
print(f"classification|     {cf:6.1%}      |     {cc:6.1%}")
print()
print("same perturbation size; only the gap distribution differs")

Output from a real run (Python 3.9.6, seed 11):

eps = 0.01  (stability threshold 2*eps = 0.02)
 
task          | top-1 flip rate | at-risk inputs (g2 < 2*eps)
retrieval     |      78.4%      |     100.0%
classification|       0.0%      |       0.0%
 
same perturbation size; only the gap distribution differs

One caveat the cell also demonstrates: with this deliberately extreme gap contrast, the classification side never flips — the paper's real classification inputs flip a few percent of the time because real gaps are not infinite. The at-risk column is the paper's real separation: retrieval leaves 100 percent of queries below the threshold, classification none.

The second cell makes the bound concrete on an actual quantization grid: a weight matrix quantized with per-group scales, the largest realized score displacement, and which candidates sit inside the resulting contender set.

import random
from math import ceil
 
# CELL 2: block-quantization rounding-error bound -- where the 2x bound bites
random.seed(23)
 
# a weight matrix: out=64 channels, in=256 inputs, group size 128
OUT, IN, GROUP, BITS = 64, 256, 128, 4
 
W = [[random.gauss(0.0, 0.08) for _ in range(IN)] for _ in range(OUT)]
X = [random.gauss(0.0, 1.0) for _ in range(IN)]     # one typical query input
 
def quantize(w, bits, group):
    step = lambda mx: 2 * mx / (2**bits - 1)
    q = [[0.0] * len(w[0]) for _ in w]
    max_round = 0.0                                   # largest per-score disp basis
    for r, row in enumerate(w):
        for g0 in range(0, len(row), group):
            g = row[g0:g0 + group]
            mx = max(abs(v) for v in g)
            s = step(mx)
            for j in range(len(g)):
                q[r][g0 + j] = round(g[j] / s) * s
                max_round = max(max_round, abs(q[r][g0 + j] - g[j]))
    return q, max_round
 
Wq, werr = quantize(W, BITS, GROUP)
 
def scores(M):
    return [sum(M[i][j] * X[j] for j in range(IN)) for i in range(OUT)]
 
z, zq = scores(W), scores(Wq)
disp = [abs(a - b) for a, b in zip(z, zq)]
eps = max(disp)
 
order = sorted(range(OUT), key=lambda i: -z[i])
g2 = z[order[0]] - z[order[1]]
violators = [i for i in range(1, OUT) if z[order[0]] - z[order[i]] < 2 * eps]
 
print(f"max per-weight rounding error (group_{GROUP}, {BITS} bits): {werr:.5f}")
print(f"largest realized score displacement eps:        {eps:.5f}")
print(f"stability threshold 2*eps:                     {2 * eps:.5f}")
print()
print(f"top-1 score           {z[order[0]]:.4f}")
print(f"top-2 score           {z[order[1]]:.4f}")
print(f"top-1/top-2 gap g2:   {g2:.5f}")
print()
print(f"contender set |C_eps| (candidates within 2*eps of top-1): {len(violators)}")
print(f"separation ratio g2 / 2*eps: {g2 / (2 * eps):.3f}  "
      f"({'stable' if g2 >= 2 * eps else 'NOT stable'})")
print()
print("the bound assumes |displacement| <= eps on every score;")
print("with a per-channel / per-group scale this holds exactly, and the")
print("criterion reduces to comparing one measured gap against 2*eps")

Output from a real run (Python 3.9.6, seed 23):

max per-weight rounding error (group_128, 4 bits): 0.02315
largest realized score displacement eps:        0.42153
stability threshold 2*eps:                     0.84307
 
top-1 score           2.5917
top-2 score           2.3443
top-1/top-2 gap g2:   0.24734
 
contender set |C_eps| (candidates within 2*eps of top-1): 4
separation ratio g2 / 2*eps: 0.293  (NOT stable)
 
the bound assumes |displacement| <= eps on every score;
with a per-channel / per-group scale this holds exactly, and the
criterion reduces to comparing one measured gap against 2*eps

Two readings. The separation ratio of 0.293 is squarely in the paper's retrieval regime (median 0.081) — nowhere near the threshold and therefore flippable. And the gap between the per-weight rounding error (0.02) and the score-level displacement (0.42) is the honest scale problem: eps at the output is an accumulated, input-dependent quantity, which is exactly why the paper measures it rather than deriving it.

The third cell mirrors the allocation economics: at a fixed budget of 3.5 bits per weight, does spending the extra half-bit on the gap-moving layers buy a disproportionate share of the full extra bit's benefit?

import random
 
# CELL 3: selective-precision economics -- gap-moving layers vs a uniform extra bit
random.seed(31)
 
L = 12               # linear layers
Q = 400              # queries
N = 200              # candidates
 
# Each layer, quantized at 3 bits, displaces every query-document score by an
# independent error uniform in [-e_l, e_l]. A few layers are far more
# gap-sensitive than the rest (paper finding: sensitivity is concentrated).
E3 = [0.080 if l in (2, 5, 8) else 0.003 for l in range(L)]
E4 = [e / 8 for e in E3]          # one extra bit: 1/8 the rounding error
 
def flip_rate(bits_per_layer):
    flips = 0
    for _ in range(Q):
        # retrieval-like scores: top items nearly tied
        z = [random.uniform(0.85, 1.0) for _ in range(N)]
        d = [sum(random.uniform(-e, e) for e in
                 (E3[l] if b == 3 else E4[l] for l, b in
                  enumerate(bits_per_layer))) for _ in range(N)]
        zp = [a + b for a, b in zip(z, d)]
        if zp.index(max(zp)) != z.index(max(z)):
            flips += 1
    return flips / Q
 
# rank layers by sensitivity / parameter count (here: equal sizes, so by e_l),
# give the extra bit to the half that moves the gap most -> avg 3.5 bits
ranked = sorted(range(L), key=lambda l: -E3[l])
gap_guided = [4 if l in ranked[:L // 2] else 3 for l in range(L)]
 
r_w3 = flip_rate([3] * L)
r_gap = flip_rate(gap_guided)
r_w4 = flip_rate([4] * L)
 
benefit_full = r_w3 - r_w4
benefit_gap  = r_w3 - r_gap
capture = benefit_gap / benefit_full
 
print("allocation                   | avg bits | top-1 flip rate")
print(f"uniform W3                   |   3.00   |     {r_w3:5.1%}")
print(f"gap-guided (6 layers at 4b)  |   3.50   |     {r_gap:5.1%}")
print(f"uniform W4                   |   4.00   |     {r_w4:5.1%}")
print()
print(f"full extra bit's benefit (fewer flips):  {benefit_full:5.1%}")
print(f"gap-guided captures:                    {capture:5.1%} of it")
print("for 50% of the extra bit's memory cost")

Output from a real run (Python 3.9.6, seed 31):

allocation                   | avg bits | top-1 flip rate
uniform W3                   |   3.00   |     97.0%
gap-guided (6 layers at 4b)  |   3.50   |     89.5%
uniform W4                   |   4.00   |     86.0%
 
full extra bit's benefit (fewer flips):  11.0%
gap-guided captures:                    68.2% of it
for 50% of the extra bit's memory cost

The toy lands inside the paper's measured band: gap-guided allocation at half the extra bit's cost captures 68.2 percent of the full bit's benefit, against the paper's 60-73 percent. The parameter that drives it is the concentration of sensitivity: when three of twelve layers dominate the score displacement, the extra half-bit buys almost what a full bit buys everywhere; make the sensitivity spread uniform and the capture collapses toward what reconstruction-error allocation buys (the paper's 40-48 percent). The method's value is empirical — it measures where the concentration actually is, per model, with L+1 cheap label-free passes.

5. Anti-hype: what "14-46 percent" is, and what it is not

The numbers are against the FP model, not against ground truth. A "top-1 change" means the quantized model returns a different first result than the FP model would — not necessarily a worse one. The paper measures this distinction: judged against BEIR relevance labels, 66 percent of flips whose FP top-1 was a relevant document replace it with a non-relevant one at W4 (77 percent at W3), and on CLIP, where each query has one correct image, 26-30 percent of changed results lose it while 13-23 percent gain it¹. So the damage is majority-harmful but not uniformly so, and a downstream system matters even for the neutral flips: a different retrieved document yields a different generated answer, and two servers running the same model at different precisions answer the same query differently. The paper's own framing of "undetected" is a benchmarking-practice claim — Recall@k and nDCG@10 on a fixed corpus can hide a 24.6 percent top-1 flip rate behind a 1.3 percent metric dip — and the correct response is to report ranking-change rates alongside aggregate metrics, not to treat every flip as an accuracy loss.

Strong quantizers reduce but do not remove this. The paper repeats the measurement with GPTQ (with and without activation ordering), AWQ, HQQ and AdaRound: all five reduce the top-1 change rate — GPTQ with activation ordering takes ViT-L/16 W4 from 42.1 to 29.6 percent and the best text cases to 14.4-14.5 percent — and none approaches the stability threshold on retrieval¹. The strongest single datapoint: GPTQ on ViT-L/16 restores classification accuracy from 55.5 to 88.6 percent and cuts the classification change rate eleven-fold, while retrieval on the same model still changes most of its top-1 results. If your mental model is "a good quantizer fixes PTQ," retrieval is the counterexample.

The bound is only as clean as the rounding-error model. The 2ε argument presumes a bounded, per-score displacement. The paper measures ε empirically per input (which sidesteps deriving it), but the regime matters: with per-channel or per-128-group scales the per-weight rounding error is bounded by half a grid step, and the accumulation to a score-level ε is measurable — this is the paper's setting, and activation quantization is explicitly untested (its Limitations list weight-only, untested activations, exact cosine over a finite corpus with approximate-nearest-neighbor indexes out of scope)¹. Schemes that transform weights non-uniformly — per-channel outlier splits, learned scales under clipping — make the displacement less cleanly bounded, and the conformal threshold inherits the exchangeability assumption. The structure of the criterion survives; the ε in "g_2 ≥ 2ε" is an empirical quantity per model per bit-width, not a constant of nature.

The routing fix is classification-only, and the paper says why. At W4 the at-risk fraction is 94-100 percent of retrieval queries against 9-26 percent of classification inputs¹; routing a minority only works when the minority is small, and for retrieval the per-input check correctly recovers almost nothing to certify. The retrieval-side answer is global (allocate bits), the classification-side answer is per-input (route), and mixing them up — routing retrieval queries on gap — would route nearly everything.

The question to ask of any quantized retrieval deployment this paper should provoke: what fraction of top-1 results differ from the FP model, measured on a held-out slice, and — if that number is not reported — why not?

6. Verdict

The paper's contribution divides cleanly. The condition (top-1 survives iff g_2 ≥ 2ε) is old mathematics, correctly credited to the semiorder and interval-order literature; the measurement is new and sharp — the same checkpoint flipping a few percent of classification answers and 33-46 percent of retrieval top-1 results at W4, with aggregate metrics blind to it, is a benchmarking-practice finding production systems should act on. The gap statistic is the right object: label-free, cheap, predictive across 2,118 configurations at Spearman −0.88, and it powers two fixes whose economics the paper measures rather than asserts (60-73 percent of an extra bit's benefit at half its cost; 85-93 percent of lost accuracy recovered by routing a quarter of inputs). The paper's own accounting prices the open questions — whether training can be made to encourage the gap a deployment needs is stated as the open question, activations and ANN indexes as out of scope. For anyone shipping quantized retrievers, the actionable core is one number to start reporting tomorrow: the top-1 agreement rate with the FP model, per corpus, before and after quantization.

Footnotes