MLPerf 2026: When the Software Is Worth 2.7x, What Is the Benchmark Measuring?

On April 1, 2026, MLCommons published the MLPerf Inference v6.0 results¹. Among the usual record claims sat a number that should change how anyone reads this benchmark: the same GB300 NVL72 rack that NVIDIA had submitted six months earlier — identical accelerators, identical memory, identical power footprint — delivered 2.77x higher DeepSeek-R1 server throughput, rising from 2,907 to 8,064 tokens per second per GPU between v5.1 and v6.0². No silicon changed. The entire gain came from the software stack.

That single datapoint is the cleanest natural experiment the inference industry has produced: it puts a number on what software maturity alone is worth on fixed hardware. And it forces the question this article answers: when a benchmark result bundles silicon, interconnect, and an optimizing software stack into one number, what exactly is being measured — and how do you read it without being spun? |

1. The Anchor: 2.7x on Unchanged Hardware

To recap the v6.0 result precisely, because every word matters: in the Closed division, the GB300 NVL72 system's DeepSeek-R1 server score per GPU went from 2,907 tokens/s (v5.1 debut, November 2025 round) to 8,064 tokens/s (v6.0, published April 1, 2026) — a factor of 2.77x². NVIDIA attributes >60% lower cost per token to the same rack and power budget. The same comparison on the DeepSeek-R1 offline scenario shows 1.68x, and on the older dense Llama 3.1 405B only 1.52x — the gains are scenario- and model-dependent, which is itself informative: mixture-of-experts reasoning models have the most headroom for serving-stack tricks.

The named techniques are not exotic: disaggregated prefill/decode serving via NVIDIA Dynamo, multi-token prediction (up to three tokens per forward pass), wider expert parallelism for MoE layers, kernel fusion, and KV-aware routing². Some of the strongest numbers were delivered by partner Nebius on the same hardware, not by NVIDIA alone — itself a data point that a mature ecosystem, not just a vendor kernel team, moves these curves.

The consequence for benchmark literacy is blunt: a "hardware benchmark" result is a (silicon × stack-vintage) product, and in 2025–2026 the stack term moved the product by more than a full GPU generation typically does. If you compare an NVIDIA result from round N against an AMD result from round N-1, the delta you see may be mostly calendar, not architecture. |

2. v6.1: One Results Table, Two Marketing Stories

The v6.1 round (published September 16, 2026; 30 organizations, 486 datacenter and edge results) made the scale question unavoidable³. AMD and its partner Crusoe submitted the largest system in MLPerf Inference history — 512 Instinct MI355X GPUs across 64 nodes — and it tops the aggregate throughput tables⁴⁵:

• GPT-OSS-120B, offline: ~5.75 million tokens/s (5.39M server)
• DeepSeek-R1, offline: ~2.90 million tokens/s (2.41M server)
• Crusoe reports linear scaling from 8 to 512 GPUs at over 90% of ideal⁴

NVIDIA's story in the same round is per-GPU and per-segment: GB300 NVL72 leads GPT-OSS-120B at both 72 GPUs (~1.20M tokens/s offline) and 8 GPUs, and CoreWeave's GB300 submission posted the highest per-GPU datacenter-closed throughput on GPT-OSS-120B of any silicon — 16,635 tokens/s/GPU offline⁶.

Both stories are true simultaneously. Normalizing (throughput ÷ accelerator count):

That is a ~48% per-GPU lead for GB300 on GPT-OSS-120B offline (16,670 vs 11,230 tokens/s/GPU). On DeepSeek-R1 offline the gap is wider: 9,441 vs 5,668 tokens/s/GPU (1.67x). Verdict by scale, all v6.1 Closed division:

The honest statement is: AMD wins the biggest-cluster row that exists; NVIDIA wins every smaller row, and wins per-GPU on GPT-OSS-120B and DeepSeek-R1 in this round. Neither headline is a lie; each is a frame. AMD's frame (aggregate records) flatters anyone who has — or plans to buy — 512 GPUs of interconnect to spend. NVIDIA's frame (per-GPU, per-segment leadership) flatters the dense-token-cost buyer. Raw totals are not comparable across cluster sizes because near-linear scaling (AMD reports 95% efficiency from 8 to 72 GPUs; NVIDIA's 288-GPU GB300 submission reports 99%⁵⁸) means roughly any vendor's cluster "wins" if it is simply made large enough — a linear-scaling illusion, not a silicon verdict. What actually limits cluster growth is the fabric, which we cover separately in Scale-Up vs Scale-Out: The Networking Math: bisection bandwidth, oversubscription ratios, and scale-out link budgets are what decide whether the 512th GPU adds throughput or congestion.

A note on the v6.0 round's own scale record for symmetry: NVIDIA's largest v6.0 system was 4× GB300 NVL72 racks (288 GPUs) over Quantum-X800 InfiniBand, then the largest ever submitted, reaching 2,494,310 tokens/s DeepSeek-R1 offline². The record changed hands one round later — to a cluster with 1.8× the GPUs. Records at maximum scale mostly measure who bothered to wire up the most GPUs that season. |

3. Vera Rubin's Preview: Almost 2x, Under a Different Label

v6.1 also carried NVIDIA's Vera Rubin NVL72 debut — in the preview category, not the available one. Per MLCommons rules, results are grouped into availability categories: Available (all components purchasable or rentable today), Preview (system must become Available by the next round), and RDI (research/internal)¹⁹. Preview entries are verified results, but the platform is pre-production, the software stack for it is its v1.0, and audits in each round target Available Closed-division submissions⁹.

The verified preview numbers, same 72-GPU system size on both sides⁸: Vera Rubin NVL72 posts 1,175,890 vs GB300's 596,944 tokens/s on DeepSeek-R1 server — about 1.97x per GPU, shrinking to ~1.74x offline (1,183,327 vs 679,740). NVIDIA's headline is "up to 2.5x" on DeepSeek-R1 (the best-case interactive scenario) and "up to 3.7x" on Qwen3-VL — a ratio driven by a single interactive-scenario comparison (1,306.6 vs 349.3 queries/s), all per the vendor's own readout of the MLCommons tables¹⁰.

Two category errors to refuse here. First, comparing a preview Vera Rubin delta (early silicon, first-generation stack) against audited available GB300 numbers and reading the ratio as a stable generation gap: tomorrow's Rubin software will move Rubin's numbers the way Dynamo moved Blackwell's, while GB300's stack is years ahead on its own optimization curve. Second, quoting "3.7x" as the general Vera Rubin speedup when nearly 2x on a frontier reasoning model is the honest per-GPU figure, and the bigger ratio belongs to one vision-language model in one latency-constrained scenario. Rubin at ~2x per GPU is a strong generational step — it just is not 3.7x, and the preview label means none of it is a shipping product yet. |

4. Method: How to Read a Results Table in Three Minutes

What a submission actually is, per MLCommons rules⁹¹:

• Divisions. Closed fixes the model, preprocessing, postprocessing, and accuracy targets — the apples-to-apples hardware comparison. Open allows model substitution and retraining, measuring best-unhinged-creativity instead.
• Scenarios. Datacenter submissions run Offline (raw batch throughput, no latency constraint), Server (throughput under a latency SLA), and Interactive (tighter token-rate and time-to-first-token requirements). A vendor leading one scenario may trail another — v6.0's 2.7x was server-scenario-specific.
• What is fixed: model weights, dataset, quality target, load generator, and rules. What is free: the entire serving stack — framework, parallelism scheme, quantization (calibration allowed, retraining not), batch policies, and cluster size.

A skeptical three-minute pass over any results table:

1. Normalize per accelerator before comparing anything (the table above took one division step to flip the v6.1 headlines).
2. Match scale. Compare 8-GPU rows to 8-GPU rows, 72 to 72. Cluster totals are marketing-shaped by construction.
3. Match round and category. Same-result-round, same-category (Available vs Preview), same-division rows only. Cross-round comparisons answer a different question — see below.
4. Check the scenario and the software column. The results table lists the framework used; "leader" on Offline batch throughput says nothing about your latency-bound product, and vLLM/TensorRT-LLM deltas are part of the measured system (see our vLLM vs SGLang comparison for how much the serving layer alone moves). |

5. Between-Round Deltas: The Signal Nobody Labels

If a cross-round comparison on fixed hardware isolates one variable — the stack — then round-over-round deltas are a benchmark hiding in plain sight. The v6.0→v6.1 window (roughly five months) delivered, all on unchanged silicon and reported by the submitters themselves:

• Lambda: +8.85% server and +8.79% offline GPT-OSS throughput on identical hardware — their own blog's number, not an MLCommons statistic¹¹
• AMD (ROCm): +28% offline / +38% server on 8× MI355X GPT-OSS-120B within one cycle; 72 v6.1 GPUs out-throughput 94 v6.0 GPUs⁵
• Intel: +36% server / +27% offline on the same four Arc Pro B70s, attributed to software maturation¹²
• CoreWeave: ~19.8% per-GPU gain on DeepSeek-R1 across GB200 NVL72 generations of submissions⁶

Spread across vendors and silicon, these 9–38% five-month deltas have the same shape as NVIDIA's 170% six-month delta, just at different stack maturities. Round-over-round on fixed hardware measures stack maturity — a genuinely useful number for buyers (it forecasts how much free performance your fleet gains before the next hardware refresh). It only becomes misinformation when the gain is reported as a hardware property in one breath while the baseline hardware is sold on last round's numbers in the next. |

6. The Honest Bait-and-Switch — and the Buyer's Heuristic

To be clear about legitimacy: optimizing for a benchmark under Closed-division rules is ordinary engineering. Disaggregated serving, kernel fusion, expert parallelism — these are the same techniques that speed up real traffic; a 2.7x on MLPerf's fixed dataset is not a cheat, it is the stack getting better at exactly the kind of work buyers run. MLPerf Closed division exists precisely to let vendors compete on stack quality against fixed models; that competition lowering everyone's cost per token is the system working as designed⁹.

The bait-and-switch is purely semantic: presenting (silicon × stack-vintage) products as silicon comparisons. "GB300 beats MI355X by 48%" and "MI355X delivers the highest throughput ever recorded" are both real v6.1 facts that each hide the other. The remedy is a reading discipline, not distrust of the benchmark:

• Compare same-scale, same-round, same-category, same-division rows — the only comparisons where stack vintage and cluster size cancel out.
• Distrust headline multiples that cannot be recomputed at the system level. If a press release's ratio survives dividing both sides by accelerator count and checking scenario/model labels, it is data; otherwise it is a frame.
• Treat preview deltas as directional, not comparable — early-silicon + first-generation-stack numbers will not stay still.
• Use cross-round deltas on fixed hardware deliberately, as the stack-maturity signal they are — and price refresh cycles accordingly: hardware bought today historically gains 9%–170% in measured throughput within six months without touching the racks.

The uncomfortable summary of MLPerf 2026 for hardware people: the most impressive numbers of the year — 2.7x on Blackwell, +38% on Instinct in five months — were produced by compilers, schedulers, and serving frameworks, on hardware everyone already owned. Silicon remains the envelope of what is possible. The benchmark increasingly measures how close your software stack has gotten to that envelope. Read every submission as a software result running on hardware, and the tables stop being a scoreboard and start being what they actually are: a quarterly report on stack maturity, per vendor, per scale. |

Related

• Scale-Up vs Scale-Out: The Networking Math Behind LLM Clusters — why cluster-scale throughput records are fabric questions (bisection bandwidth, oversubscription), not GPU questions.
• NVIDIA B200 vs GB200: Efficiency Benchmark — the same scale-skew trap in MLPerf Training v5.0 data.
• vLLM vs SGLang — two of the serving stacks behind these numbers, compared directly.

Sources

Author: flozi00 | Published: September 24, 2026