GPU Buying Guide for LLM Inference (2026)

Every "which GPU should I buy for LLM inference" question decomposes into three checks, in this order: does the model plus its KV cache fit in VRAM, is the memory bandwidth high enough for your token-rate target, and only then, does the compute matter. This guide covers how to read GPU specs through that lens, what the actual market tiers cost as of September 2026, and where the traps are — used data-center cards, consumer-card limits, and interconnect. The VRAM arithmetic itself is covered in our existing deep dives, so we link rather than re-derive.

1. How to Read GPU Specs for LLM Serving

Decode is memory-bandwidth-bound. At inference time, autoregressive decode reads every active model weight from VRAM once per generated token. At small batch sizes, FLOPS barely matter: tokens/s per GPU is bounded by bandwidth ÷ active weight bytes. A 70B model in BF16 is ~140 GB of bytes per token, so an H100 SXM (3.35 TB/s) has a hard ceiling near 24 tok/s per GPU at batch 1, while a B200 (8 TB/s in HGX configuration) reaches ~57 tok/s — regardless of the 4.5× FLOPS gap between them. Prefill and large-batch decode are compute-bound instead, which is where tensor-core throughput (FP8 on Hopper, native FP4 on Blackwell) earns its keep. The full roofline is derived in our inference math article.

Capacity decides what fits — and what you can batch. VRAM must hold weights plus KV cache plus activations. Quantization trades accuracy for both capacity and effective bandwidth (bytes per token), so the cheapest "GPU upgrade" is often a format change rather than a purchase — see the quantization guide. A model that fits comfortably leaves room for longer context and bigger batches, which is the real lever for cost per served token. Work the numbers with the VRAM calculator and the KV-cache deep dive before you spend anything.

So the spec-sheet reading order for serving is: VRAM capacity (hard gate) → memory bandwidth (decode speed) → NVLink/interconnect (multi-GPU scaling) → FLOPS (prefill and batch). Power draw and cooling multiply all of it and get a word below.

2. The Market Tiers and Prices (as of September 2026)

A hard honesty note first: this market has no list prices. NVIDIA publishes none for H100/H200/B200/B300; everything below is reseller listings, OEM system prices divided by GPU count, and analyst estimates re-verified across at least two independent current sources. Prices are volatile — the same H100 has spanned $25k–$40k for over a year. Figures below are converted to EUR at the September 2026 USD/EUR rate of ~1.16 ¹; treat every number as a range with a ±20% haircut for negotiating reality (provenance in ²). B200 and B300 are effectively not sold as single cards at all — they ship inside 8-GPU HGX systems ($430k–$860k per node listed), so per-GPU prices for them are system-derived ³.

Tiers in one line each. Consumer flagships (RTX 5090, 32 GB): fast decode per euro, capped at ~32B BF16 / ~70B at 4-bit per card. Workstation (RTX PRO 6000 Blackwell, 96 GB ECC): single-slot 70B–120B serving for teams. Used previous-gen data center (A100): cheap HBM capacity, deprecated silicon. Current data center (H100/H200): the production workhorses with mature ecosystem. Current flagship (B200/B300): buy when VRAM or per-rack throughput demands it — effectively a system purchase, not a card purchase ⁵.

3. Interconnect: Why Multi-GPU Wants NVLink

Tensor parallelism splits every transformer layer across GPUs, so each generated token requires an all-reduce over activations at every layer boundary. Over PCIe Gen5 x16 (~128 GB/s bidirectional) that communication serializes against a 3.35 TB/s-class memory system; over NVLink it overlaps it. Per-GPU NVLink bandwidth by tier, per NVIDIA's documentation: NVLink 3 on A100 at 600 GB/s, NVLink 4 on H100/H200 at 900 GB/s, NVLink 5 on B200/B300 at 1.8 TB/s — versus ~64 GB/s per direction for PCIe Gen5 x16 ^{6 7}. NVLink also needs the SXM baseboard (or NVL bridge pairs); loose PCIe cards talk to each other through the host root complex at PCIe speeds.

Practical consequences: a single H100-class card on PCIe serves fine — TP=1 has no inter-GPU traffic. Two or four GPUs with NVLink bridges still scale well. Eight-way TP inside an HGX SXM node is where NVSwitch matters. And cross-node TP is so communication-bound that pipeline-parallel layouts deliver up to 4.5× higher throughput on 16 cross-node H100s, per a ScitiX AI study — the reason "more GPUs" beyond one node needs a layout rethink, not just a bigger fabric ⁸.

4. Risks by Tier

Used data-center cards (A100, ex-cloud H100s). Data-center parts have no consumer warranty; NVIDIA's enterprise coverage follows the original system integrator, and mining-style 24/7 history is invisible — ask for running-hours and thermal logs where possible. A100s are discontinued (CDW lists the PNY A100 80GB PCIe as "Discontinued" at a frozen $13.2k last-listed price), so spares and repair channels thin out over the hardware's remaining life ³. SXM modules require a matching baseboard — a cheap SXM card without one is a paperweight. Fake or stripped engineering samples circulate on secondary marketplaces; buy from resellers with return policies.

Consumer cards. 32 GB caps model choice before anything else does. No MIG partitioning (an H100 SXM splits into up to 7 isolated ~10 GB instances (7×1g.10gb; the 18 GB 1g.18gb profile belongs to the H200) — useful for many-small-model serving). The RTX 5090 has no ECC on GDDR7; the RTX PRO 6000 does — relevant for long unattended runs where silent bit-flips corrupt weights. Consumer cards have no NVLink; multi-GPU rigs pay PCIe overhead and don't scale TP well past 2 GPUs. Two-slot board designs and 575 W TBP also mean a single consumer-grade PSU and case can host at most one or two.

System realities at the top. A B200 draws ~1,000 W and a B300 ~1,400 W per GPU; an 8-GPU B300 node is listed at ~14.5 kW — liquid cooling territory, and a real facility cost on top of the sticker price ³.

5. Decision Table by Workload

Verdict

Buy the tier your capacity math dictates, not the tier with the best headline FLOPS: run the inference calculator with your real model, context, and concurrency first. In 2026 the surprising answers are that a €2k consumer card decodes a 30B model as fast as a €30k H100 (bandwidth ratio, not FLOPS), that the H200 is often the production sweet spot (141 GB at barely above H100 street prices), and that B200/B300 purchases are facility projects, not card upgrades. When in doubt, rent first — specialist-cloud H100 rates near $3/GPU-hour let you measure your real utilization before you commit capital ³.

Further Reading

• LLM Inference Math: From Theory to Hardware — the bandwidth-roofline derivation behind every token-rate claim above.
• How Much VRAM Does an LLM Need? — weight and KV-cache sizing before you shop.
• LLM Quantization — the cheapest capacity upgrade is a format change, not a card.
• KV Cache Explained — why long-context workloads should shop capacity-first.
• Inference Calculator — run your real model, context, and batch through the math.