AMD MI300X/MI350/MI400: More Memory, But Does the Math Win? — A Critical Silicon View of AMD Instinct vs NVIDIA

AMD's Instinct line has one marketing message, repeated every generation: more HBM and more bandwidth per GPU than NVIDIA. That claim is true — every number below is pinned from AMD's own datasheets. And yet most deployed tokens run on CUDA hardware. This is the math-first explanation of both facts: where the silicon wins on paper, where it loses in production, and why each loss has a named, sourced cause.

The Silicon Line-Up: Datasheet Numbers, Not Slides

Three caveats. The MI355X's headline OCP-FP8 figure (5,033.2 TFLOPS) is dense per AMD's datasheet³ — with structured sparsity it doubles to 10,066.4, and likewise the FP16 row (2,516.6 dense, 5,033.2 with sparsity). NVIDIA follows the same convention (the B200's 2,250 TFLOPS dense BF16 is the honest figure). And the MI400 numbers are vendor-claimed and internally inconsistent: the product page says 23.3 TB/s, the Helios brochure 19.6 TB/s per GPU⁴.

The computed ratios

MI355X vs H200: 2.04x the memory, 1.67x the bandwidth, 2.54x the FP8 peak (both figures dense). Against the B200 — its actual generational peer — the MI355X is 1.60x the capacity but 1.01x the bandwidth. AMD's lead is over Hopper-class parts, not Blackwell.

For the MI455X: 3.06x H200 capacity, 4.08x its bandwidth (2.45x the B200's) — the largest paper lead AMD has ever claimed, and the first generation where bandwidth ratio exceeds capacity ratio. Pressure-tested below.

Why Raw Memory Does Not Win Inference

Reason 1: Quantization support is not symmetric

Memory capacity only counts if you can fill it with a model, and production models are quantized. vLLM's own hardware-compatibility table is blunt: on AMD GPUs, AWQ, GPTQ, the Marlin fast kernels (GPTQ/AWQ/FP8/FP4) and bitsandbytes are all unsupported; FP8 W8A8 and GGUF are the supported paths⁷. On NVIDIA, all of those are ticked, with hand-tuned kernels per format.

SGLang's platform table tells the same story: FP8 (w8a8) works on MI300X/MI325X/MI350 via Aiter/Triton, AWQ runs only through a Triton dequantize path instead of the JIT-compiled CUDA kernel, GPTQ was removed in favor of the NVIDIA-only gptq_marlin (docs.sglang.io, as of Sept 2026), and the AMD-specific quark_int4fp8_moe path (online INT4-to-FP8 MoE) exists precisely because the mature INT4 stack is missing on CDNA⁸.

The arithmetic consequence: the smallest well-supported footprint on MI300X is FP8 (1 byte per weight); the same operator on H100/H200 runs W4A16 with Marlin kernels, and MXFP4/NVFP4 is first-class on B200. A 70B model occupies roughly 70 GB at FP8 versus roughly 40 GB at 4-bit — more concurrent sequences, more KV cache, or a smaller GPU bill. The 2.4x memory lead shrinks to about 1.4x once only the NVIDIA side can quantize harder — the ratio of model-copies-per-GPU: (192 GB ÷ 70 GB) ÷ (80 GB ÷ 40 GB) = 2.74 ÷ 2.00 = 1.37 ⁷⁸.

70B model footprint, weights only:
FP8 weights            ~70 GB   (vLLM-supported on AMD and NVIDIA)
4-bit (AWQ/GPTQ-INT4)  ~40 GB   (mature Marlin kernels: NVIDIA only)
 
Llama-70B GQA KV cache: ~0.33 MB/token (BF16) -> a single 128k-token
context adds ~43 GB of KV on top of the weights.

Reason 2: Compute-vs-memory balance (the roofline)

Decoding is memory-bound: per-sequence ceiling equals bandwidth over model bytes. For a 70B BF16 model (140 GB), that is about 24 tokens/s on an H100 versus 38 on a MI300X — a 1.58x edge tracking the bandwidth ratio exactly. Batch-1 memory-bound decoding is the one regime where the datasheet advantage converts one-for-one into throughput.

But prefill and large-batch decode are compute-bound, and there AMD's balance is worse: the arithmetic-intensity crossover (FP8 TFLOPS per TB/s) is about 493 for MI300X versus 591 for H100 (629 for the MI355X, over 1,000 claimed for the MI455X). MI300X tilts toward memory (FP16 peak 1.32x H100 against 2.40x capacity), so the advantage evaporates where interactive serving spends half its time: prompt processing. It is also why MI355X's 2x FP16/FP8 peak over MI300X mattered more than its +50% memory.

At rack level, an 8xMI300X node (42.4 TB/s aggregate) gives a sharded 671B FP8 DeepSeek-class model a decode ceiling near 63 tokens/s versus 40 on 8xH100 and 57 on 8xH200 — the capacity advantage means the model shards onto fewer GPUs with room for KV cache, the honest version of AMD's win.

Reason 3: The scale-up domain is 8 GPUs, not 72

The MI300X/MI355X platform connects 8 GPUs via seven Infinity Fabric links per GPU (128 GB/s per link on MI300X, ~153.6 GB/s on MI355X)¹³. NVLink 4 gives each H100 900 GB/s bidirectional, NVLink 5 each B200 1.8 TB/s — and, crucially, every GPU pair in a 72-GPU NVL72 rack talks at NVLink speed through the switch fabric⁹.

That domain size is a serving constraint, not a footnote. TP efficiency degrades across fabric boundaries: TP8 fits one 8-GPU domain on either vendor, but TP16 (or low-latency expert parallelism) spans domains — at 8 on MI300X/MI355X, with scale-out crossing Ethernet/InfiniBand at far lower per-pair bandwidth, versus 72 on NVL72. MLPerf reflects it: AMD's strongest results are 8-GPU nodes, while NVIDIA submits 72-GPU NVL72 systems running frontier-class benchmarks (DeepSeek-R1, 671B) in-domain¹⁰. For frontier serving the effective TP domain is the product — and "8" is what AMD shipped against "72" until Helios.

Software Stack Reality vs the Benchmark Record

ROCm's release cadence has improved dramatically — Day-0 model support is now a stated AMD goal³ — but cadence was never the bottleneck; kernel maturity is. The vLLM/SGLang tables above are ground truth: the fast kernels that make NVIDIA's smaller memories workable do not exist on CDNA, and AMD compensates with its own paths (Quark, Aiter, quark_int4fp8_moe)⁷⁸.

The benchmark record shows a real but narrow gap closing. AMD's first-ever MLPerf Training submission came in v5.0 (June 2025), with MI350-series results following in v5.1 (November 2025)¹¹ — years after NVIDIA. MLPerf Inference v5.1 (September 2025) had AMD submitting MI355X systems with strong Llama-2-70B server numbers alongside NVIDIA's record entries¹⁰. Third-party production benchmarks (SemiAnalysis/InferenceX, MangoBoost LLMBoost) show MI300X competitive-to-ahead on batch memory-bound serving and behind on prefill-heavy, low-latency interactive workloads — third-party measurements, not vendor claims. An independent arXiv evaluation found strong memory-bound performance with software immaturity as the recurring cost¹².

The pattern: whenever AMD's win requires only the memory system, it wins. Whenever it requires kernel depth, quantization breadth, or a large TP domain, NVIDIA wins. The benchmarks are not biased; they measure the software.

MI400/Helios: Do the Promises Survive the Physics?

As of September 2026, the MI400 series is announced but not generally shipping: MI455X and Helios launched 2026-07-23, with OpenAI expecting racks online from Q4 2026⁵. Everything below is vendor-claimed.

The claimed specs: 432 GB HBM4, 19.6 TB/s per GPU (23.3 TB/s on the product page), 40.3 PFLOPS MXFP4 / 20.1 PFLOPS FP8, 72 GPUs per rack via UALoE at 3.6 TB/s per GPU, 31 TB HBM4 per rack⁴⁵. Recomputed: 19.6 TB/s is a 4.08x bandwidth lead over H200 and 2.45x over B200 — and unlike MI355X vs B200 (a 1.0x tie), the bandwidth ratio now exceeds the capacity ratio. It is the first AMD generation whose paper roofline favors both decode (4.08x bandwidth) and prefill (claimed FP8 crossover ~1,026 ops/byte), and whose 72-GPU UALoE domain answers NVL72's topology directly.

Credit where the math checks out: Helios' 72-GPU fabric addresses the domain-size problem (Reason 3) head-on, and HBM4's bandwidth per GB genuinely beats anything B200 ships. Skepticism: (a) AMD's actual 2026-2027 competitor is Rubin with NVLink 6, not B200 — no third-party numbers exist for that pairing yet; (b) the 23.3-vs-19.6 TB/s discrepancy between AMD's own pages says the rounding has started; (c) the quantization gap (Reason 1) closes with kernel engineering, not a new memory subsystem; (d) "30% more inference tokens per dollar"⁵ is a per-dollar figure against an unnamed baseline — it prices the competitive dynamics in. Verdict: the Helios silicon math survives scrutiny; the delivery schedule, kernel depth, and Rubin's response do not exist yet, and an untestable claim is not yet a fact.

Decision Table: When AMD Actually Wins

Compute the fits (weights + 15% overhead, before KV cache) and the verdict falls out mechanically.

The "More VRAM = Better" Trap, As One Worked Example

One scenario compresses every failure mode above — a 405B FP8 model (405 GB weights), served interactively:

• On 8xH100: 640 GB minus 405 GB weights leaves 235 GB across eight GPUs — ~29 GB/GPU KV headroom. It fits, barely, via TP8 in one NVLink domain.
• On 8xMI300X: 1,536 GB minus the same 405 GB leaves 1,131 GB — 141 GB/GPU KV headroom, 4.8x more. The 2.4x memory advantage is real.

And yet the NVIDIA system typically wins delivered tokens: (a) every request's prefill half is compute-bound, where B200-class FP8 (~4,500 TFLOPS) delivers about 1.7x MI300X (2,614.9); (b) at 4-bit with Marlin kernels the NVIDIA side reclaims most of the capacity gap⁷; (c) traffic spilling past TP8 crosses nodes on both systems, but NVL72 keeps four more TP8-width domains coherent at NVLink speed before the network tax⁹. The 2.4x memory advantage is a property of the memory subsystem; deployed throughput is a property of the whole stack. Marketing shows the first and lets you assume the second — the ratios above are the antidote.

None of this makes Instinct a bad product — it makes it a specialist: the only shipping hardware that puts a 100–670B model on one board with KV-cache room, and from MI455X onward possibly the first credible rack-scale answer NVIDIA has had. Buy it for the workloads the table marks as AMD wins; budget for the software realities of Reasons 1–3 everywhere else.

Article first published: September 24, 2026 Author: flozi00

Related Reading

• NVIDIA NVLink Deep Dive — the fabric on the other side
• The KV Cache: Bit-Exact Memory Math — every capacity claim here, derived