LLM Inference Economics: Hardware Limits, Costs & Cloud Break-Even

Serving LLMs is a sandwich of physics and economics. The physics — memory bandwidth, FLOPS, VRAM capacity — caps how many tokens a given GPU can produce. The economics — acquisition prices, power, cloud rates, API list prices — decides whether those tokens are cheaper on your own hardware or on someone else's. This page walks through both, with interactive calculators wired to the same roofline math as the inference calculator.

Getting started: the two resources that limit you

Memory: the batch-1 bottleneck

At low concurrency, decode is memory-bandwidth-bound. Generating one token means streaming every active model parameter from VRAM once:

$\text{tok/s} \approx \frac{\text{memory bandwidth}}{\text{active parameter bytes}}$

Two more memory limits stack on top:

• Capacity: weights + KV cache must fit in VRAM. A 70B model at BF16 needs ~140 GB before a single token of context — no consumer card can hold it.
• Amortization ceiling: batching spreads the weight-read cost over more requests. But every additional sequence also grows the KV cache, so at some point VRAM is full and the batch stops growing.

Compute: the ceiling you hit at scale

The same GPU also has a hard FLOPS roof. Each token costs ~2 FLOPs per active parameter (multiply + accumulate), independent of quantization:

$\text{tok/s}_{\text{max}} = \frac{\text{FLOPS}}{2 \times \text{active parameters}}$

Unlike bandwidth, this ceiling is batch-independent: the weights are read once per decode step regardless of how many requests share them. Batching moves you toward it; nothing moves you past it.

When does the limit shift from memory to compute?

Per request, throughput stops improving when the aggregate bandwidth ceiling reaches the compute ceiling — the break-even batch B*. Below B*, more requests ride along on the same weight reads for free; at B*, the tensor cores saturate; beyond B*, every extra request dilutes everyone's speed. For a dense 32B model on an RTX PRO 6000, B* is around 280; for MoE models with small active parameter counts, B* can be in the thousands. The break-even analyzer computes B* for your exact combination.

Economics and reality

Which models fit which hardware?

Fit is a pure capacity question first: total parameters × bytes per parameter must leave room for KV cache. The matrix below shows it live for your chosen precision and GPU count — green means ≥15% headroom, amber means it technically fits but leaves almost no KV budget, red means it doesn't fit. Quantization shifts entire rows: FP8 halves every weight footprint, NVFP4 quarters it.

How expensive is the hardware?

Rule-of-thumb figures (Sept 2026, editable in the calculator): a single RTX PRO 6000 is ~$8.5k street, H100 ~$25k, H200 ~$28k, B200 ~$34k, MI300X ~$12k. A complete 8-GPU server multiplies card cost by roughly 1.8 for the chassis, CPUs, and networking. And the sticker price is only the entry ticket: power (a 700 W H100 at €0.25/kWh is ~$130/mo per card), housing, staff, and the cost of capital roughly double the monthly bill over 36 months of amortization.

How many tokens do you get out of the hardware?

Token yield = throughput at the operating point (the smaller of B* and the VRAM-feasible batch) × wall-clock. An RTX PRO 6000 serving Qwen3-30B-A3B at FP8 at 70% utilization produces on the order of 47 billion tokens per month; a B300 serving a 6B-active MoE produces roughly 5× that. The calculator computes this from the roofline — including the honesty that utilization, not peak throughput, is what most fleets get wrong.

When is cloud cheaper — and when is own hardware cheaper?

Three cost lines matter: cloud rent (per GPU-hour, linear in time), own hardware (mostly fixed per month, linear in tokens), and APIs (pure per-token). The crossover logic:

• Low, spiky utilization (< ~40%) → cloud wins. You pay only for hours used; owning idle silicon is the most expensive way to serve nothing.
• Steady high utilization (> ~60%) with a model that fits → own hardware wins by a factor of 3–10 on raw cost per token. A $8.5k card that outperforms a $2.20/h cloud GPU pays for itself in a few months at 24/7.
• Sporadic workloads, many different models, no ops team → APIs win despite the per-token premium, because the premium buys elasticity and zero operations.
• Compliance boundaries (ISO 27001, C5, DSGVO) → own hardware or dedicated hosting is often the only option, regardless of price; data must not leave the perimeter.

The calculator computes the crossover utilization and the break-even months for your concrete workload.

Open-weight models vs proprietary APIs

The uncomfortable truth for API vendors: at scale, tokens from open weights are dramatically cheaper. A 96 GB card serving a 30B-class MoE costs a few cents per million tokens in electricity and amortization; GPT-5 mini charges $0.25/$2.00 per million in/out. The table below compares each open-weight model with the proprietary class it realistically competes with — same quality class, not token-identical. Two caveats keep it honest: your time to operate the stack isn't priced in at API rates, and quality-per-token still differs — the comparison tells you what a quality class costs, not that the models are interchangeable.

Limitations

• All throughput numbers are roofline ceilings; real serving stacks achieve 50–80% of them.
• Prices are editable defaults (Sept 2026 street/list ballparks) — plug in your actual contract rates.
• The open-vs-proprietary table assigns quality classes; benchmarks move fast, verify against your own evals.
• Serving-stack overhead (vLLM/SGLang scheduling, CPU preprocessing, networking) is not modeled.