LLM Inference Calculator & Break-Even Analyzer
Two interactive tools for planning LLM serving hardware. The planner estimates VRAM fit, decode throughput, and time-to-first-token for a concrete model + GPU + precision + batch configuration. The break-even analyzer answers the deeper capacity question: at which batch size does decode stop being bandwidth-limited, at which point is it compute-limited, and what is the maximum sequence length you can serve at that batch.
All numbers are theoretical ceilings derived from vendor specs and roofline math — real deployments typically land at 60–80% of these values. The inference math guide explains the underlying formulas.
1. The planner: capacity, throughput, latency
Interactive Estimator
LLM inference planner beta
Total parameters drive VRAM capacity; active parameters drive decode speed (MoE only fetches routed experts per token). MLA and hybrid-attention models use compressed KV-cache math.
Capacity check (Single GPU)
- Model weights (total)
- 64 GB
- Active weights per token
- 64 GB
- KV cache per token
- 0.26 MB
- Total KV cache (5,120 tok × 1)
- 1.34 GB
- Activations
- 0.8 GB
- Runtime overhead
- 1.2 GB
- Total VRAM needed
- 67.34 GB
- Available VRAM
- 96 GB
- Headroom
- 28.66 GB
- Max batch @ current context
- ~22
- Max context @ batch 1
- ~114,452 tok
Roofline alignment
- GPU ops:byte (BF16 (2 B))279.02 ops/byte
- Base intensity (head_dim/2)64 ops/byte
- Effective intensity (× batch)64 ops/byte
- Gap215.02 ops/byte
Memory-bound: raising batch size lifts effective intensity because more tokens share each active-weight fetch.
Latency snapshot
- Decode throughput
- 21 tok/s
- Time per token
- 47.62 ms
- Throughput limit
- Memory
- Time to first token (4,096 tok)
- 745.96 ms
- Total request time
- 49.51 s
- Memory-limited ceiling
- 21 tok/s
- Compute-limited ceiling
- 2,929.69 tok/s
Decode ≈ min(aggregate bandwidth ÷ per-step bytes, FLOPS(BF16) ÷ 2·active-params, per-layer sync latency). Sync model: 0 sync(s)/layer × PCIe Gen5 x16 latency, scaled by kernel efficiency — TP+EP on PCIe fabrics is sync-bound, not bandwidth-bound.TTFT = max(total weight stream, linear + quadratic attention FLOPs). MoE capacity still needs all 32B params in VRAM.
2. Break-even: when does batching stop helping?
At batch size 1, decode is bandwidth-bound: generating one token means streaming every active parameter from HBM once, so tokens/s ≈ memory bandwidth ÷ active weight bytes. Each additional request in the batch rides along on those same weight reads — aggregate throughput climbs almost linearly while per-request speed stays roughly constant.
This cannot continue forever. Two things end the free lunch:
- Compute crossover — the total FLOPs the batch needs per step exceed what the GPU's tensor cores can deliver in one memory round. Beyond that batch size, the GPU is compute-bound and per-request speed degrades.
- MoE expert-coverage saturation — in Mixture-of-Experts models, larger batches keep touching more distinct experts until all routed experts are read once per step. After that point, extra requests add weight traffic again instead of amortizing it.
The break-even batch B* is the smaller of the two — the last batch size where batching still buys you per-request speed. Serve at B* for maximum interactive responsiveness; serve above it only for raw aggregate throughput.
Break-even Analyzer
Decode break-even calculator beta
At low batch, decode is bandwidth-bound: every request re-reads the active weights, so tokens/s scales with the batch. Past the break-even batch B*, weight reads amortize no further (MoE expert coverage is complete, or the compute roofline is hit) — per-request speed saturates and then only degrades. This panel finds B* and the largest context that fits there.
Break-even analysis
B* ≈ 140 needs more KV memory than this setup holds (VRAM fits ~56 sequences at 2,048 tokens). The setup never reaches the compute-bound regime — batch as high as VRAM allows and per-request speed keeps rising toward the B* value.
Decode throughput vs batch size
Aggregate tok/s = min(bandwidth ceiling, compute ceiling). The bandwidth ceiling stays flat while MoE expert coverage is incomplete (each request adds its own expert reads), then grows linearly once every routed expert is read once per step. B* is the first batch where the bandwidth ceiling reaches the flat compute ceiling — beyond it, decode is compute-bound and per-request speed degrades. The VRAM cap line shows where the batch stops fitting at a 2,048-token context.
How the break-even is computed
Both calculators share the same decode model:
- Bandwidth ceiling (aggregate tok/s) = effective bandwidth × GPUs ÷ per-step bytes, where per-step bytes grow with the batch until MoE expert coverage is complete (
min(activeBytes × batch, totalWeightBytes)). - Compute ceiling (aggregate tok/s) = effective FLOPS × GPUs ÷ (2 × active parameter bytes) — batch-independent, because the weights are read once per step regardless of batch.
- B* = the batch where the two ceilings intersect (for dense models this is very large; for MoE models expert saturation often hits first), i.e. where the workload transitions from memory-bound to compute-bound.
- Max sequence length at B* = (total VRAM − weights − runtime overhead) ÷ (KV bytes per token × B*), capped at the model's context window.
The analyzer also reports the VRAM-feasible maximum batch at a ~1k-token context. When B* is larger than that, the setup never actually reaches the compute-bound regime — batch as high as VRAM allows and per-request speed keeps improving toward the B* ceiling.
Limitations
- No continuous-batching arrival dynamics: B* is a steady-state ceiling, not a scheduling policy.
- Attention decode FLOPs (growing with context length) are not in the compute ceiling; at very long contexts the real crossover arrives earlier.
- Multi-GPU assumes tensor parallelism with perfect bandwidth scaling; PCIe fabrics add sync latency the planner models but the curve does not.