MLPerf Inference 2026: What a Result Says About Hardware and Software

MLPerf Inference measures submitted systems, including accelerators, networking and software. Its results can compare complete deployments under the benchmark's rules. They do not by themselves separate a chip's contribution from the serving stack. The MLCommons results and rules should accompany any headline drawn from the score table.

A substantial gain on the same accelerator family

NVIDIA's analysis of MLPerf Inference v6.0 reports DeepSeek-R1 Server throughput on GB300 NVL72 rising from 2,907 to 8,064 tokens/s per GPU between its v5.1 and v6.0 submissions. The quotient is about 2.77. The same vendor table reports 1.68× for DeepSeek-R1 Offline and 1.52× for Llama 3.1 405B Server. These are per-GPU figures calculated from complete submissions; NVIDIA explicitly notes that per-chip throughput is not MLPerf's primary metric.

The comparison is strong evidence that a benchmark score is not a permanent property of a GPU model. NVIDIA attributes the v6.0 improvement to changes in its serving software, including kernel improvements and attention data parallelism. It is not a controlled software ablation: the submissions are from different rounds, and the public comparison does not hold every configuration variable constant. The precise contribution of each software change cannot be inferred from the two scores. NVIDIA's statement about lower cost per token also relies on its infrastructure and power assumptions; the throughput ratio alone does not measure energy or cost.

Scale changes the headline

MLCommons' account of v6.1 records a 512-accelerator submission (IDs 6.1-0026 and 6.1-0027) and almost 5.8 million tokens/s on GPT-OSS-120B Offline for one of them. That is a total-system throughput result. It cannot rank a single accelerator against another vendor's smaller system.

Dividing total throughput by accelerator count gives a useful descriptive number, but it does not remove differences in workload, scenario, latency constraint, quantization, interconnect or scale. Two systems can have different throughput per accelerator simply because their batch sizes and communication costs differ. For a deployment comparison, first match the model, scenario, quality target, result round, availability category and approximate accelerator count. Then inspect the full system description and latency requirements. A per-accelerator ratio is an additional calculation, not a hardware-only benchmark.

Preview is a separate category

The v6.1 MLCommons discussion says that some leading results used a Preview Vera Rubin system. The v6.1 supplemental document identifies a Nebius preview submission using 36 Rubin GPUs, alongside available GB300 systems at several sizes. A preview score should retain its category and configuration when quoted. Vendor-reported “up to” speedups across scenarios are not a universal Rubin-to-GB300 multiplier.

A practical reading checklist

1. Open the MLCommons results dashboard and identify the submission IDs.
2. Match benchmark version, Closed/Open division, model, scenario, accuracy target and availability category.
3. Record the total throughput, accelerator count, system topology and latency constraints. Compare similar scales where possible.
4. Use reported system power and actual operating costs for performance-per-watt or cost-per-token claims. Do not infer them from throughput alone.
5. For a claim that software caused a speedup, look for a controlled comparison or an ablation. Cross-round scores show the combined change in the submitted system.

The main lesson of the 2026 rounds is that both software changes and deployment scale can move a score substantially. A result becomes useful for a purchasing decision only when its complete configuration matches the intended workload closely enough.