Leaderboard Margins vs Hidden Model Selection: Which Gains Survive k Secret Variants?

When a lab posts a two-point win on a public LLM leaderboard, the most expensive assumption in the room is that the score you see is the score they ran. It usually is not. A provider privately evaluates some number of variants, ships the best one, and the board displays that champion next to a fixed comparator. The published margin is a maximum over a private family, not a single draw. A September 2026 statistics preprint, "How Sensitive Are LLM Leaderboard Claims to Hidden Model Selection?" (Chen Yang, Xianyang Zhang and Jun Chen, arXiv:2609.28177)¹, takes that mechanism seriously and derives what it does to leaderboard claims — and the answer is more precise, and more useful, than the usual "leaderboards are noisy" shrug.

The claim under test is not the one you think

The paper formalizes a setup with three fixed objects: a provider privately scores k sibling variants, submits the best, and the board shows the submitted score next to a public comparator, with per-item correctness available for both displayed models. The statistical target is the provider-level null: no hidden variant truly beats the comparator. Rejection certifies that some variant of the provider is genuinely better — not necessarily the submitted one². The losing siblings, their count k, and how correlated they are never become public. So the audit cannot estimate the search size; it can only ask the reverse question: given the margin I observe, how many hidden variants could this claim absorb and still stand?

That reverse question is what makes the paper useful. Instead of a verdict ("lab X cheated"), you get a sensitivity curve: for each assumed lower bound on within-family correlation, the largest number of hidden variants the margin still certifies. The curve turns one unobservable — private search size — into a family of explicit assumptions a reader can argue with.

The Gaussian margin model, in one act

Scores are asymptotically Gaussian and couple through item-level correctness. The working model has three components: item difficulty shared by every model (which cancels in any margin), a family component shared by the siblings (which is what makes them alike), and idiosyncratic noise (which is what selecting a maximum tends to inflate). Standardized, variant scores correlate pairwise at rho_w, variants correlate with the comparator at rho_b. Margins — variant minus comparator — then have variance 2*(1-rho_b) and pairwise correlation r = (1+rho_w-2rho_b) / (2-2rho_b)³.

The worst case for an auditor is the one where every hidden sibling is exactly as good as the comparator: worse siblings contribute less to the maximum, better ones make the null false. At that least-favorable configuration, the tail of the maximum has a classical one-factor Dunnett form (Dunnett, 1955) — the k-th power of a conditional tail, averaged over one shared Gaussian factor⁴. Out of that comes the certificate: for observed standardized margin z, the curve reports the largest hidden multiplicity the claim survives at significance level alpha. The paper illustrates with k-bar(0.56) = 17: the claim is certified for up to 17 hidden variants if every sibling pair correlates at 0.56 or more. At 27 variants it is not, whatever the provider actually did.

Watching the maximum eat significance

The intuition is cheap to simulate. Generate k Gaussian variants with within-family correlation rho_w under the null (all means equal to the comparator), take the maximum, and compare it against the naive single-test 5 percent threshold (z = 1.645). At k=1 the false-certification rate is exactly 5 percent. It does not stay there:

import numpy as np, math
from math import erf, sqrt, log
 
def ppf(p, lo=-10, hi=10):        # normal quantile by bisection, no scipy needed
    for _ in range(80):
        mid = (lo + hi) / 2
        if 0.5*(1 + erf(mid/sqrt(2))) < p: lo = mid
        else: hi = mid
    return (lo + hi) / 2
 
rho_b = 0.5                       # comparator correlation (paper default)
naive = ppf(0.95)                 # k=1 threshold: 1.645
def sim(k, rho_w, rho_b=0.5, n=400_000, rng=None):
    rng = rng or np.random.default_rng(7)
    r = (1 + rho_w - 2*rho_b) / (2 - 2*rho_b)   # margin correlation
    G = rng.standard_normal(n)                 # shared family factor
    E = rng.standard_normal((n, k))            # idiosyncratic noise
    m = np.sqrt(r)*G[:, None] + np.sqrt(1-r)*E # unit-variance margins
    return m.max(axis=1) * sqrt(2*(1-rho_b))   # rescale to margin units
 
print(f"naive (k=1) 5% threshold: z = {naive:.3f}")
print("false-certification rate if k hidden variants exist, rho_w = 0.56:")
for k in [1, 5, 27, 100]:
    print(f"  k={k:3d}: {(sim(k, 0.56) > naive).mean():.3f}")
print("threshold the auditor needs at k=27 (95th pct of the max):")
for rho_w in [0.92, 0.71, 0.56, 0.42]:
    q95 = np.quantile(sim(27, rho_w, n=1_000_000,
                       rng=np.random.default_rng(42)), 0.95)
    z = q95 / sqrt(2*(1-rho_b))
    p1 = 0.5*(1 + erf(z/sqrt(2)))              # single-variant tail at that z
    ne = log(0.95) / log(p1)                   # independent-equivalent count
    print(f"  rho_w={rho_w:.2f}: needed z = {z:.2f}  (behaves like ~{ne:.0f} free tries)")

naive (k=1) 5% threshold: z = 1.645
false-certification rate if k hidden variants exist, rho_w = 0.56:
  k=  1: 0.050
  k=  5: 0.158
  k= 27: 0.347
  k=100: 0.507
threshold the auditor needs at k=27 (95th pct of the max):
  rho_w=0.92: needed z = 2.16  (behaves like ~3 free tries)
  rho_w=0.71: needed z = 2.54  (behaves like ~9 free tries)
  rho_w=0.56: needed z = 2.68  (behaves like ~14 free tries)
  rho_w=0.42: needed z = 2.78  (behaves like ~19 free tries)

Three things fall out. First, the erosion is fast but not catastrophic: 27 hidden variants at correlation 0.56 behave like roughly 14 independent tries, not 27 — the family component ties the siblings together, so they buy the provider fewer free lottery tickets than independent tries would⁵. Second, the required margin is systematic, not a cliff: at the paper’s anchors, an adjacent-rank claim needs z around 2.7 rather than 1.645 at k=27 and correlation floors 0.42–0.56, a tolerance most adjacent leaderboard pairs simply do not have. Third, the naive 5 percent test degrades in a specific, quantifiable direction: at k=100 and rho_w=0.56, half of pure-noise champions still clear the untouched bar. The paper’s own effective-multiplicity grid at k=27 spans n_eff = 5–19 across its illustrative scenarios — the simulation here lands inside that band.

0.90, 0.46, 0.92: the correlation must match the ranked score

Here is the part most coverage will garble. The certificate is indexed by the correlation of the score the board actually ranks by — and that is not the correlation you would naively measure. The Open LLM Leaderboard ranks by the equal-weight mean of six per-benchmark accuracies. Under within-benchmark item resampling, a benchmark’s share of the composite’s sampling variance scales with 1/n_b — the number of items — while a correlation pooled over all items weights each benchmark by its item share. The two estimators answer different questions⁶.

The paper’s worked example makes it concrete: in a controlled Qwen2.5-1.5B full fine-tune family evaluated on MMLU + GSM8K, GSM8K supplies only 8.6 percent of the items but drives about 91 percent of the two-task score’s sampling variance. Pooling over items emphasizes MMLU and reports a median within-family correlation of 0.894; score-matched item resampling emphasizes GSM8K and reports 0.459. Same models, same items, same run — different estimand. In 4 of 12 hand-curated observational families the two estimates even disagree on which family is more correlated.

The 0.92 is the third punchline: if you resample MMLU’s 57 subjects as whole blocks instead of items, while keeping the item-weighted score, the worked family’s score-matched median jumps from 0.459 to 0.921⁷. The pooled-versus-score-matched contrast is therefore conditional on the sampling model — "what varies between hypothetical re-evaluations?" decides which correlation is even relevant. None of these three numbers is the correlation; each is correct under a stated resampling scheme, and only the score-matched one under item resampling matches what this leaderboard ranks by.

The audit: 394 adjacent-rank claims, 391 without support

With the model in hand, the authors audit the Open LLM Leaderboard’s per-item correctness matrix: 395 models, 28,659 items, six benchmarks (Polo et al., 2024)⁸. They take all 394 adjacent-rank cross-provider pairs — the "model A beats model B by X" claims a board exists to make — and test each as a standardized margin with a paired standard error under independent within-benchmark item resampling, at alpha = 0.05. The pairs overlap and are not independent, so the counts are descriptive. No selection correction has entered yet. The naive column certifies 3 of 394. The Gaussian column — which prices in k=27 hidden variants at the audited correlation floors — certifies zero. That is the "391 lack statistical support" headline, and it is ordinary sampling uncertainty, before hidden selection is even accounted for⁹.

Read the medians and the number stops being shocking: the median adjacent margin is z = 0.17 and the 95th percentile is z = 0.78. Most adjacent leaderboard ranks were never separated from zero by their own items. Of the three claims that do pass the naive test, two fall to "not margin-certified" once selection at the audited correlation floors is accounted for, and one becomes model-insufficient — the assumed correlation floor falls outside the margin’s domain for a near-clone comparator, a statement about the model’s applicability, not about the lab¹⁰. Bonferroni, which needs no correlation assumption at all, still supports that last one.

What survives, and what a reader should conclude

The anti-hype point runs both directions, and losing the second half is how this paper gets misquoted. It is not "leaderboards are useless." What survives is precise:

1. Aggregate movement is real. The authors state aggregate decisions on the audit are stable — a board can still tell you which tier a model belongs in.
2. Large margins are fine. A gap of several points on a composite with thousands of items clears z = 2.7 with room to spare; "our model leads by three points" claims are not the ones at risk. What dies is the adjacent-rank comparison — the two-place swap your team slacks about.
3. Between two labs, ask for the pair, not the ranks. The certificate regime needs per-item data for both models, a paired standard error, and an assumed correlation floor. If two labs’ models sit one rank apart, the honest summary is "indistinguishable at item level" until either a bigger gap or independent confirmation data appears.
4. A declared variant count does not fix anything by itself. The authors explicitly note a self-reported count alone does not verify that the candidate set was fixed before benchmark feedback — prospective disclosure needs the protocol, not the number¹¹.

For a board operator the paper’s ask is modest: request a candidate count plus a record of how the candidate set was fixed, and publish a finite-range sensitivity curve with stated assumptions and applicability status. For a reader, the compact heuristic this article walks away with: at alpha = 0.05 and rho_w around 0.5–0.6, an adjacent-rank claim needs an effective z around 2.5–2.8 to survive twenty-odd hidden variants, while the median adjacent margin on a dense board sits near 0.2. The gap between those two numbers is the hidden-selection problem — not a reason to ignore leaderboards, but a reason to demand more from one-place wins.

What to remember

• A published leaderboard margin is a maximum over k privately evaluated variants; the provider-level null asks whether any variant truly beats the comparator.
• At rho_w = 0.56, the naive 5 percent test false-certifies 15.8 percent at k=5, 34.7 percent at k=27, 50.7 percent at k=100 — but correlation makes 27 variants behave like only ~14 independent tries (paper grid: n_eff 5–19).
• The required z at k=27 rises from 1.645 to 2.16 / 2.54 / 2.68 / 2.78 as the within-family floor drops from 0.92 to 0.42.
• The 0.90/0.46/0.92 triple is one family under three estimands: pooled-item vs score-matched item resampling vs subject-block resampling — only the score-matched one matches what the board ranks by.
• 391 of 394 adjacent-rank Open LLM Leaderboard claims lack statistical support before selection; of the 3 that survive the naive test, selection accounting withdraws 2 and pushes 1 out of the model’s domain.