Leaderboard Margins vs Hidden Model Selection: Which Gains Survive k Secret Variants?

When a lab posts a two-point win on a public LLM leaderboard, hidden model selection is a possibility worth testing. If a provider privately evaluates k variants and submits the best-scoring one, the published margin is a maximum over that private family, not a single draw. The paper treats this as a sensitivity scenario; it does not establish that any audited provider actually followed that process. A September 2026 statistics preprint, "How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?" (Chen Yang, Xianyang Zhang and Jun Chen, arXiv:2609.28177)¹, takes that mechanism seriously and derives what it does to leaderboard claims — and the answer is more precise, and more useful, than the usual "leaderboards are noisy" shrug.

The claim under test is not the one you think

The paper formalizes a setup with three fixed objects: a provider privately scores k sibling variants, submits the best, and the board shows the submitted score next to a public comparator, with per-item correctness available for both displayed models. The statistical target is the provider-level null: no hidden variant truly beats the comparator. Rejection certifies that some variant of the provider is genuinely better — not necessarily the submitted one². In the audited public snapshots, the losing siblings, their count k, and their correlations are not available. So the audit cannot estimate the search size; it can only ask the reverse question: given the margin I observe, how many hidden variants could this claim absorb and still stand?

That reverse question is what makes the paper useful. Instead of a verdict ("lab X cheated"), you get a sensitivity curve: for each assumed lower bound on within-family correlation, the largest number of hidden variants the margin still certifies. The curve turns one unobservable — private search size — into a family of explicit assumptions a reader can argue with.

The Gaussian margin model, in one act

Scores are asymptotically Gaussian and couple through item-level correctness. The working model has three components: item difficulty shared by every model (which cancels in any margin), a family component shared by the siblings (which is what makes them alike), and idiosyncratic noise (which is what selecting a maximum tends to inflate). Standardized, variant scores correlate pairwise at rho_w, variants correlate with the comparator at rho_b. Margins — variant minus comparator — then have variance 2*(1-rho_b) and pairwise correlation r = (1+rho_w-2rho_b) / (2-2rho_b)³.

The worst case for an auditor is the one where every hidden sibling is exactly as good as the comparator: worse siblings contribute less to the maximum, better ones make the null false. At that least-favorable configuration, the tail of the maximum has a classical one-factor Dunnett form (Dunnett, 1955) — the k-th power of a conditional tail, averaged over one shared Gaussian factor⁴. Out of that comes the certificate: for observed standardized margin z, the curve reports the largest hidden multiplicity the claim survives at significance level alpha. The paper illustrates with k-bar(0.56) = 17: the claim is certified for up to 17 hidden variants if every sibling pair correlates at 0.56 or more. At 27 variants it is not certified under that assumed floor; the curve says nothing about how many variants the provider actually tested.

Watching the maximum eat significance

The intuition is cheap to simulate. Generate k Gaussian variants with within-family correlation rho_w under the null (all means equal to the comparator), take the maximum, and compare it against the naive single-test 5 percent threshold (z = 1.645). At k=1 the false-certification rate is exactly 5 percent. It does not stay there:

import numpy as np, math
from math import erf, sqrt, log
 
def ppf(p, lo=-10, hi=10):        # normal quantile by bisection, no scipy needed
    for _ in range(80):
        mid = (lo + hi) / 2
        if 0.5*(1 + erf(mid/sqrt(2))) < p: lo = mid
        else: hi = mid
    return (lo + hi) / 2
 
rho_b = 0.5                       # comparator correlation (paper default)
naive = ppf(0.95)                 # k=1 threshold: 1.645
def sim(k, rho_w, rho_b=0.5, n=400_000, rng=None):
    rng = rng or np.random.default_rng(7)
    r = (1 + rho_w - 2*rho_b) / (2 - 2*rho_b)   # margin correlation
    G = rng.standard_normal(n)                 # shared family factor
    E = rng.standard_normal((n, k))            # idiosyncratic noise
    m = np.sqrt(r)*G[:, None] + np.sqrt(1-r)*E # unit-variance margins
    return m.max(axis=1) * sqrt(2*(1-rho_b))   # rescale to margin units
 
print(f"naive (k=1) 5% threshold: z = {naive:.3f}")
print("false-certification rate if k hidden variants exist, rho_w = 0.56:")
for k in [1, 5, 27, 100]:
    print(f"  k={k:3d}: {(sim(k, 0.56) > naive).mean():.3f}")
print("threshold the auditor needs at k=27 (95th pct of the max):")
for rho_w in [0.92, 0.71, 0.56, 0.42]:
    q95 = np.quantile(sim(27, rho_w, n=1_000_000,
                       rng=np.random.default_rng(42)), 0.95)
    z = q95 / sqrt(2*(1-rho_b))
    p1 = 0.5*(1 + erf(z/sqrt(2)))              # single-variant tail at that z
    ne = log(0.95) / log(p1)                   # independent-equivalent count
    print(f"  rho_w={rho_w:.2f}: needed z = {z:.2f}  (behaves like ~{ne:.0f} free tries)")

naive (k=1) 5% threshold: z = 1.645
false-certification rate if k hidden variants exist, rho_w = 0.56:
  k=  1: 0.050
  k=  5: 0.158
  k= 27: 0.347
  k=100: 0.507
threshold the auditor needs at k=27 (95th pct of the max):
  rho_w=0.92: needed z = 2.16  (behaves like ~3 free tries)
  rho_w=0.71: needed z = 2.54  (behaves like ~9 free tries)
  rho_w=0.56: needed z = 2.68  (behaves like ~14 free tries)
  rho_w=0.42: needed z = 2.78  (behaves like ~19 free tries)

Three things fall out. First, the erosion is fast but not catastrophic: 27 hidden variants at correlation 0.56 behave like roughly 14 independent tries, not 27 — the family component ties the siblings together, so they buy the provider fewer free lottery tickets than independent tries would⁵. Second, the required margin is systematic, not a cliff: at the paper’s anchors, an adjacent-rank claim needs z around 2.7 rather than 1.645 at k=27 and correlation floors 0.42–0.56, a tolerance most adjacent leaderboard pairs simply do not have. Third, the naive 5 percent test degrades in a specific, quantifiable direction: at k=100 and rho_w=0.56, half of pure-noise champions still clear the untouched bar. The paper’s own effective-multiplicity grid at k=27 spans n_eff = 5–19 across its illustrative scenarios — the simulation here lands inside that band.

0.90, 0.46, 0.92: the correlation must match the ranked score

Here is the part most coverage will garble. The certificate is indexed by the correlation of the score the board actually ranks by — and that is not the correlation you would naively measure. The Open LLM Leaderboard ranks by the equal-weight mean of six per-benchmark accuracies. Under within-benchmark item resampling, a benchmark’s share of the composite’s sampling variance scales with 1/n_b — the number of items — while a correlation pooled over all items weights each benchmark by its item share. The two estimators answer different questions⁶.

The paper’s worked example makes it concrete: in a controlled Qwen2.5-1.5B full fine-tune family evaluated on MMLU + GSM8K, GSM8K supplies only 8.6 percent of the items but drives about 91 percent of the two-task score’s sampling variance. Pooling over items emphasizes MMLU and reports a median within-family correlation of 0.894; score-matched item resampling emphasizes GSM8K and reports 0.459. Same models, same items, same run — different estimand. In 4 of 12 hand-curated observational families the two estimates even disagree on which family is more correlated.

The 0.92 is the third punchline: if you resample MMLU’s 57 subjects as whole blocks instead of items, while keeping the item-weighted score, the worked family’s score-matched median jumps from 0.459 to 0.921⁷. The pooled-versus-score-matched contrast is therefore conditional on the sampling model — "what varies between hypothetical re-evaluations?" decides which correlation is even relevant. None of these three numbers is the correlation; each targets a stated resampling scheme. Both score-matched estimates preserve the ranked score, but item and subject resampling posit different sources of sampling variation. The paper uses item resampling for its primary audit.

The audit: 394 adjacent-rank claims, 391 without support

With the model in hand, the authors audit the Open LLM Leaderboard’s per-item correctness matrix: 395 models, 28,659 items, six benchmarks (Polo et al., 2024)⁸. They take all 394 adjacent-rank cross-provider pairs — the "model A beats model B by X" claims a board exists to make — and test each as a standardized margin with a paired standard error under independent within-benchmark item resampling, at alpha = 0.05. The pairs overlap and are not independent, so the counts are descriptive. No selection correction has entered yet. The naive column certifies 3 of 394. The Gaussian column — which prices in k=27 hidden variants at the audited correlation floors — certifies zero. That is the "391 lack statistical support" headline, and it is ordinary sampling uncertainty, before hidden selection is even accounted for⁹.

Read the medians and the number stops being shocking: the median adjacent margin is z = 0.17 and the 95th percentile is z = 0.78. Most adjacent leaderboard ranks were never separated from zero by their own items. Of the three claims that do pass the naive test, two fall to "not margin-certified" once selection at the audited correlation floors is accounted for, and one becomes model-insufficient — the assumed correlation floor falls outside the margin’s domain for a near-clone comparator, a statement about the model’s applicability, not about the lab¹⁰. Bonferroni, which needs no correlation assumption at all, still supports that last one.

What survives, and what a reader should conclude

The anti-hype point runs both directions, and losing the second half is how this paper gets misquoted. It is not "leaderboards are useless." What survives is precise:

1. Aggregate movement is real. The authors state aggregate decisions on the audit are stable — a board can still tell you which tier a model belongs in.
2. Large standardized margins can survive. Whether a gap of several score points clears z = 2.7 depends on its paired standard error and the selection assumptions. The audit finds most adjacent-rank margins unresolved, while its v2 example with z = 3.01 survives the displayed corrections¹.
3. Between two labs, ask for the pair, not the ranks. The certificate regime needs per-item data for both models, a paired standard error, and an assumed correlation floor. If two labs’ models sit one rank apart, the honest summary is "indistinguishable at item level" until either a bigger gap or independent confirmation data appears.
4. A declared variant count does not fix anything by itself. The authors explicitly note a self-reported count alone does not verify that the candidate set was fixed before benchmark feedback — prospective disclosure needs the protocol, not the number¹¹.

For a board operator the paper’s ask is modest: request a candidate count plus a record of how the candidate set was fixed, and publish a finite-range sensitivity curve with stated assumptions and applicability status. For a reader, the compact heuristic this article walks away with: at alpha = 0.05 and rho_w around 0.5–0.6, an adjacent-rank claim needs an effective z around 2.5–2.8 to survive twenty-odd hidden variants, while the median adjacent margin on a dense board sits near 0.2. The gap between those two numbers is mostly the sampling-uncertainty problem already visible before selection; hidden selection raises the threshold further for the few otherwise resolvable claims — not a reason to ignore leaderboards, but a reason to demand more from one-place wins.

What to remember

• Under the paper’s hidden-selection scenario, a published margin is a maximum over k privately evaluated variants; the provider-level null asks whether any variant truly beats the comparator.
• At rho_w = 0.56, the naive 5 percent test false-certifies 15.8 percent at k=5, 34.7 percent at k=27, 50.7 percent at k=100 — but correlation makes 27 variants behave like only ~14 independent tries (paper grid: n_eff 5–19).
• The required z at k=27 rises from 1.645 to 2.16 / 2.54 / 2.68 / 2.78 as the within-family floor drops from 0.92 to 0.42.
• The 0.90/0.46/0.92 triple is one family under three estimands: pooled-item vs score-matched item resampling vs subject-block resampling — both score-matched estimates preserve the ranked score under different resampling assumptions.
• 391 of 394 adjacent-rank Open LLM Leaderboard claims lack statistical support before selection; of the 3 that survive the naive test, selection accounting withdraws 2 and pushes 1 out of the model’s domain.