GPU Buying Guide for LLM Inference (2026)

Every GPU selection for LLM inference starts with model and KV-cache fit, then measures the required decode and prefill rates on the intended stack. This guide covers the relevant manufacturer specifications, interconnect, used-hardware risks and how to compare current purchase quotes. The VRAM arithmetic itself is covered in the linked deep dives.

1. How to Read GPU Specs for LLM Serving

Small-batch dense decode is often bandwidth-limited. Its idealized weight-only ceiling is memory bandwidth divided by weight bytes, provided the weights and KV cache fit. A 70B bf16 checkpoint needs about 140 GB before KV, so it does not fit on one 80 GB H100 SXM; dividing H100 bandwidth by 140 GB would describe an infeasible single-GPU setup. A 32.76B bf16 text checkpoint needs about 65.5 GB: the idealized weight-only ceilings are ~51 tok/s on H100 SXM (3.35 TB/s) and ~118 tok/s on HGX B200 (7.7 TB/s), before KV traffic or other limits. Prefill and larger batches can hit the compute ceiling. See the inference math article.

Capacity decides what fits — and what you can batch. VRAM must hold weights plus KV cache plus activations. Quantization trades accuracy for both capacity and effective bandwidth (bytes per token), so the cheapest "GPU upgrade" is often a format change rather than a purchase — see the quantization guide. A model that fits comfortably leaves room for longer context and bigger batches, which is the real lever for cost per served token. Work the numbers with the VRAM calculator and the KV-cache deep dive before you spend anything.

Read the sheet in this order: VRAM capacity (fit), memory bandwidth (small-batch decode), tensor throughput (prefill and batching), interconnect (multi-GPU communication), then power and cooling. Which limit binds depends on the workload.

2. Hardware tiers and comparable quotes

The table names a specific SKU where bandwidth differs by form factor. The figures are peak manufacturer specifications, not measured serving rates. A quote must identify the exact module, chassis, interconnect and warranty; HGX B200/B300 are normally quoted as systems, so a system price divided by eight is not a standalone card price.

The RTX 5090 and workstation RTX PRO 6000 have the same peak memory bandwidth, but the latter has three times the memory capacity. The HGX B300 row follows NVIDIA's dedicated Blackwell Ultra datasheet; some NVIDIA reference-architecture material lists different maximum B300 memory/bandwidth figures, so check the exact offered system. Sources: RTX 5090, RTX PRO 6000, H200, Blackwell Ultra.

For an economic comparison, request current system quotes and divide total ownership cost by measured useful tokens over the service life. Include chassis, networking, power, cooling, support, utilization and resale value. Historic reseller listings are not reliable purchase prices.

3. Interconnect: Why Multi-GPU Wants NVLink

Tensor parallelism splits layer operations across GPUs and requires communication during each forward pass. PCIe Gen5 x16 offers about 64 GB/s per direction (128 GB/s combined); NVIDIA quotes 600 GB/s for A100 NVLink 3, 900 GB/s for H100/H200 NVLink 4, and 1.8 TB/s for B200/B300 NVLink 5 as aggregate GPU-to-GPU link bandwidth. These link totals are not automatically usable for a particular all-reduce: topology, message size and software matter ¹. HGX SXM systems provide NVSwitch connectivity; particular PCIe NVL cards can use compatible bridges.

TP=1 has no inter-GPU communication. For multi-GPU serving, compare measured latency and throughput of tensor, pipeline and expert parallel layouts on the offered topology. A study reported up to 4.5× throughput for a pipeline-oriented layout over its cross-node TP baseline on 16 H100s; that is a workload-specific result, not a general multiplier ².

4. Risks by Tier

Used data-center cards (A100, ex-cloud H100s). Check the seller's warranty, return rights, firmware, thermal condition and module identity. SXM modules require a compatible baseboard and cooling; an SXM module alone is not a drop-in PCIe card. Ask for burn-in results and a working system configuration before treating a low module price as a bargain.

Consumer cards. The RTX 5090 has 32 GB GDDR7, no NVLink and a 575 W total graphics power rating. Its 32 GB cannot hold a 70B model even at ideal 4-bit payload size (35 GB before scales and KV). The RTX PRO 6000 Workstation Edition has 96 GB GDDR7 ECC. Multi-GPU PCIe performance depends on topology and runtime; verify power supply and cooling for the exact cards and chassis.

System realities at the top. NVIDIA's Blackwell Ultra datasheet lists HGX B300 GPU TDP configurable up to 1,100 W per GPU; actual node draw includes CPUs, networking, cooling and power conversion. Obtain the complete system's rated and measured power requirements from the OEM before planning facilities ³.

5. Decision Table by Workload

Verdict

Size the model, KV cache and runtime reserves with the inference calculator, then benchmark candidate systems at the target latency and concurrency. Compare current full-system quotes and operating costs for the measured workload. A roofline ceiling or an old listing cannot establish that one GPU is the cheapest choice.

Further Reading

• LLM Inference Math: From Theory to Hardware — the bandwidth-roofline derivation behind every token-rate claim above.
• How Much VRAM Does an LLM Need? — weight and KV-cache sizing before you shop.
• LLM Quantization — the cheapest capacity upgrade is a format change, not a card.
• KV Cache Explained — why long-context workloads should shop capacity-first.
• Inference Calculator — run your real model, context, and batch through the math.