AMD MI300X/MI350/MI400: More Memory, But Does the Math Win? — A Critical Silicon View of AMD Instinct vs NVIDIA

AMD's Instinct GPUs offer substantial HBM capacity and bandwidth. The table below compares selected AMD and NVIDIA products using vendor specifications. Capacity, bandwidth, compute, quantization kernels, and interconnect topology each matter, but none alone establishes delivered tokens per second for a particular workload.

Published Vendor Specifications

The MI355X's OCP-FP8 figure (5,033.2 TFLOPS) is dense per AMD's product specifications³; structured sparsity doubles that figure, as it does for its FP16 matrix rate. The NVIDIA B200 entries use dense rates too. AMD's newer MI455X product page specifies 23.3 TB/s, while an earlier Helios brochure uses 19.6 TB/s per GPU and labels its specifications engineering projections subject to change. The calculations below explicitly use that earlier figure as an illustrative specification scenario⁴.

The computed ratios

MI355X vs H200: 2.04x the memory, 1.67x the bandwidth, 2.54x the FP8 peak (both figures dense). Against an HGX B200 at its up to 8 TB/s specification, the MI355X is 1.60x the capacity and approximately tied on peak bandwidth. These are specification ratios, not measured serving throughput.

Using the brochure's 19.6 TB/s, the MI455X has 3.06x H200 capacity and 4.08x its bandwidth (2.45x the HGX B200's up-to-8 TB/s figure). The product page's 23.3 TB/s would produce different ratios.

Why Memory Capacity Alone Does Not Decide Inference

Reason 1: Quantization support is not symmetric

The available quantization path depends on the engine, version, GPU generation, and checkpoint. vLLM's published compatibility table lists its AWQ, GPTQ, Marlin and bitsandbytes implementations as unavailable on AMD GPUs, while listing FP8 W8A8 and GGUF. However, vLLM's current ROCm platform code also advertises awq, gptq, quark, and mxfp4 paths; its awq_marlin entry is mapped to ordinary AWQ⁷. A compatibility-table cross is therefore not proof that every 4-bit model is impossible on ROCm.

SGLang's table lists AWQ on AMD via Triton dequantization, FP8 via Aiter or Triton, and MXFP4 on CDNA3/CDNA4 via Aiter. Its gptq_marlin and awq_marlin kernels are CUDA-only, while quark_int4fp8_moe is an AMD-specific option⁸. These are different execution paths; the table does not quantify their speed or quality on a given model.

For the same 70B checkpoint, roughly 70 GB of FP8 weights or around 40 GB of 4-bit weights is a useful planning estimate before metadata, activations, and KV cache. The 4-bit option must be checked for the chosen engine and GPU. Comparing FP8 on one vendor with 4-bit on the other also changes the model representation, so a capacity ratio alone cannot establish a performance winner⁷⁸.

70B model footprint, weights only:
FP8 weights            ~70 GB   (plus scales and runtime memory)
4-bit weights          ~40 GB   (format and backend dependent)
 
Llama-70B GQA KV cache: ~0.33 MB/token (BF16) -> a single 128k-token
context adds ~43 GB of KV on top of the weights.

Reason 2: Compute-vs-memory balance (the roofline)

In an idealized batch-1, weight-bandwidth-bound decode, bandwidth divided by weight bytes gives a ceiling. For a 70B BF16 model (140 GB), that is about 24 tokens/s on an H100 versus 38 on a MI300X — a 1.58x ratio tracking the bandwidth ratio exactly. KV reads, compute, communication and achieved bandwidth can make real decoding slower or change the bottleneck.

Prefill and large-batch decode may be compute-bound. The theoretical FP8 compute-to-bandwidth ratios are about 493 operations per byte for MI300X and 591 for H100 (629 for MI355X). MI300X has 1.32x H100's dense FP16 peak, less than its 1.58x bandwidth ratio. That suggests a smaller relative advantage in compute-heavy phases, not that it must be slower. Actual time-to-first-token depends on model architecture, kernel implementation, prompt length, and batch size.

At node level, eight MI300X GPUs provide 42.4 TB/s of aggregate peak bandwidth. Dividing this by 671 GB of FP8 weights gives an idealized 63 tokens/s ceiling for a sharded 671B-parameter model, before communication, KV traffic, and other overhead. Eight H100 GPUs have only 640 GB total memory, so that uncompressed 671-GB weight representation does not fit there; dividing their aggregate bandwidth by 671 GB would not describe a runnable configuration.

Reason 3: The scale-up domain is 8 GPUs, not 72

MI300X/MI355X baseboards connect eight GPUs through Infinity Fabric (the MI300X platform sheet describes seven peer links per GPU; MI355X specifies seven 153.6-GB/s links)¹³. NVLink 4 provides up to 900 GB/s aggregate bidirectional bandwidth per H100, and NVLink 5 up to 1.8 TB/s per B200/GB200-class GPU. NVL72 is a 72-GPU NVLink domain, but that per-GPU figure is shared across communications; it is not dedicated bandwidth for every pair simultaneously⁹.

Domain size can constrain tensor and expert parallelism. TP8 fits within one eight-GPU baseboard; TP16 on MI300X/MI355X spans nodes and therefore uses the scale-out network. An NVL72 installation can keep more GPUs in one NVLink domain⁹. This is a topology advantage for some communication-heavy layouts, not a measured token-throughput comparison. The actual result depends on the model's parallel layout and traffic pattern.

Software Stack Reality vs the Benchmark Record

AMD states that it targets day-zero support for optimized models³. Current vLLM and SGLang documentation lists both CUDA-specific and ROCm-specific quantization paths⁷⁸. Which implementation is faster cannot be decided from a support matrix; it needs a benchmark with the same model, precision, workload, and latency target.

AMD submitted MLPerf Training results in v5.0 (June 2025), followed by MI350-series results in v5.1 (November 2025)¹⁰. MLPerf Inference v5.1 (September 2025) includes AMD MI355X systems¹¹. These results establish tested configurations, not a universal NVIDIA-versus-AMD ordering across workloads. A separate MI300X evaluation reports strong memory-bound behavior and software-stack limitations in its tested setup¹².

For a purchase or deployment decision, compare full-system results at the required precision and service-level target. Peak specifications identify plausible configurations; they do not settle that comparison.

MI400/Helios: Do the Promises Survive the Physics?

As of September 2026, the MI400 series is announced but not generally shipping: MI455X and Helios launched 2026-07-23, with OpenAI expecting racks online from Q4 2026⁵. Everything below is vendor-claimed.

The announced specifications include 432 GB HBM4, 20.1 PFLOPS FP8, and a 72-GPU Helios rack with 31 TB HBM4 and UALoE at a stated 3.6 TB/s per GPU⁴⁵. The earlier brochure's 19.6 TB/s is 4.08x H200's published bandwidth and 2.45x HGX B200's up-to figure; the newer product-page value is higher. Those are peak specification ratios. Neither the ratios nor a computed FP8 roofline balance establish delivered decode or prefill throughput.

The stated 72-GPU Helios fabric addresses the scale-up-domain difference, and both published AMD bandwidth figures exceed HGX B200's published per-GPU peak⁴. The earlier 19.6-TB/s figure is an engineering projection, while the newer product page says 23.3 TB/s; neither is a measured delivered bandwidth. AMD's "up to 30% more inference tokens per dollar" claim also depends on its undisclosed comparison setup⁵. Independent, matched workloads are needed to judge delivered performance and cost.

Configuration Table: What the Numbers Establish

The fits below use decimal GB, 15% illustrative runtime allowance on weights, and an explicit KV reserve where stated. They establish capacity constraints, not a throughput winner.

The "More VRAM = Better" Trap, As One Worked Example

Consider a 405B FP8 model with approximately 405 GB of raw weights, before runtime allowances and KV cache:

• On 8×H100: 640 GB minus 405 GB of raw weights leaves 235 GB across eight GPUs, about 29 GB/GPU for all remaining uses, including KV cache.
• On 8×MI300X: 1,536 GB minus 405 GB leaves 1,131 GB, about 141 GB/GPU for all remaining uses. This is about 4.8× the residual capacity in this particular calculation.

This calculation proves a capacity difference, not a throughput result. A comparison against H100 must use H100 compute and bandwidth; substituting B200 or NVL72 figures would change the hardware under comparison. Request mix, quantization, kernels, and communication must be benchmarked on the two actual systems before claiming a winner.

The practical use of these figures is to rule configurations in or out by memory before measuring the ones that fit. A 671B FP8 model needs multiple MI300X GPUs, such as an eight-GPU baseboard; it cannot fit in the memory of one MI300X. The Helios figures describe an announced rack-scale configuration and still require workload measurements.

Article first published: September 24, 2026 Author: flozi00

Related Reading

• NVIDIA NVLink Deep Dive — the fabric on the other side
• The KV Cache: Bit-Exact Memory Math — every capacity claim here, derived