Friendly disclaimer: flozi00 TechHub is a solo side-project next to a full-time job — personal learning notes, no official statements. Verify critical steps yourself.
Weights are only one part of memory use. After weights and runtime allowances, the remaining KV budget must cover stored tokens across concurrent requests.
Two interactive tools for planning LLM serving hardware. The planner estimates VRAM fit, decode throughput, and time-to-first-token for a concrete model + GPU + precision + batch configuration. The break-even analyzer answers the deeper capacity question: at which batch size does decode stop being bandwidth-limited, at which point is it compute-limited, and what is the maximum sequence length you can serve at that batch.
The throughput figures are idealized roofline estimates, not measured serving rates. The efficiency slider is an assumption rather than a verified universal 60–80% range. The inference math guide explains the formulas.
Total parameters drive VRAM capacity; active parameters drive decode speed (MoE only fetches routed experts per token). MLA and hybrid-attention models use compressed KV-cache math.
Capacity check (Single GPU)
Model weights (total)
65.52 GB
Active weights per token
65.52 GB
KV cache per token
0.26 MB
Total KV cache (5,120 tok × 1)
1.34 GB
Activations
0.8 GB
Runtime overhead
1.2 GB
Total VRAM needed
68.86 GB
Available VRAM
96 GB
Headroom
27.14 GB
Max batch @ current context
~21
Max context @ batch 1
~108,639 tok
Roofline alignment
GPU ops:byte (BF16 (2 B))279.02 ops/byte
Decode intensity (batch 1)1 ops/byte
Effective intensity (batch 1)1 ops/byte
Gap278.02 ops/byte
Memory-bound: raising batch size lifts effective intensity because more tokens share each active-weight fetch.
Latency snapshot
Decode throughput
20.51 tok/s
Time per token
48.75 ms
Throughput limit
Memory
Time to first token (4,096 tok)
762.62 ms
Total request time
50.69 s
Memory-limited ceiling
20.51 tok/s
Compute-limited ceiling
5,723.09 tok/s
Decode ≈ min(aggregate bandwidth ÷ per-step bytes, FLOPS(BF16) ÷ 2·active-params, per-layer sync latency). Sync model: 0 sync(s)/layer × PCIe Gen5 x16 latency, scaled by kernel efficiency — TP+EP on PCIe fabrics is sync-bound, not bandwidth-bound.TTFT = max(total weight stream, linear + quadratic attention FLOPs).MoE capacity still needs all 32.762B params in VRAM.
At small batches, weight reads can dominate decode. For a dense model in the simplified roofline model, one decode step reads the weights once for the batch: additional requests can raise aggregate throughput while reducing weight bytes per generated token. KV traffic, attention, kernels and scheduling still affect per-request speed. For MoE, different requests may route to overlapping or different experts, so the number of distinct weights read per step depends on routing.
Two modeled limits matter:
Compute crossover — the idealized bandwidth and compute ceilings meet at B*. The actual crossover also depends on KV reads, attention, kernel efficiency and communication.
MoE coverage approximation — the calculator caps per-step weight reads at total model size: min(active weight bytes × batch, total weight bytes). This assumes no overlap below the cap and complete coverage at the cap. Neither is generally true for routed experts; treat the reported saturation batch as a toy-model marker, not a measured threshold.
B* is where these two modeled ceilings meet. For a dense model with batch-independent weight reads, the formula is GPU FLOPS × bytes per parameter ÷ (2 × bandwidth), using consistent effective throughput and bandwidth. The calculator may find that B* exceeds the VRAM-feasible batch. Use the curve to explore assumptions, then benchmark latency and throughput at the target context and concurrency.
Break-even Analyzer
Decode break-even calculator beta
This simplified roofline estimates weight-read and compute ceilings as batch size changes. B* is where those modeled ceilings meet. MoE expert overlap, KV traffic, attention and scheduling can move the real crossover; benchmark your workload. The panel also estimates KV capacity at B*.
B* ≈ 280 needs more KV memory than this setup holds (VRAM fits ~53 sequences at 2,048 tokens). The modeled compute crossover is unreachable at that context. More batching may increase aggregate throughput, but per-request speed and latency must be measured..
Decode throughput vs batch size
Aggregate tok/s = min(bandwidth ceiling, compute ceiling). The MoE weight-read model assumes no expert overlap below its total-weight cap and complete coverage at the cap; the actual curve depends on routing. B* is where the modeled memory ceiling first reaches the modeled compute ceiling. The VRAM line is the capacity limit at a 2,048-token context.
Bandwidth estimate (aggregate tok/s) = batch × effective bandwidth × occupied GPU capacity ÷ modeled per-step weight bytes. For MoE, the per-step bytes are approximated by min(activeBytes × batch, totalWeightBytes).
Compute estimate (aggregate tok/s) = effective FLOPS × GPUs ÷ (2 × active parameters), ignoring attention and other operations.
B* = first modeled batch where the bandwidth estimate reaches the compute estimate. It is not a runtime scheduling setting or a verified bound for a real model.
Max sequence length at B* = (total VRAM − weights − runtime overhead) ÷ (KV bytes per token × B*), capped at the model's context window.
The analyzer also reports an estimated maximum batch by KV capacity at a ~2k-token context. Runtime allocations and activation memory can lower the actual limit. Increasing batch can reduce per-request rate even below B*, especially with MoE routing, KV traffic or scheduling overhead.
No continuous-batching arrival dynamics: B* is a steady-state ceiling, not a scheduling policy.
Attention decode FLOPs (growing with context length) are not in the compute ceiling; at very long contexts the real crossover arrives earlier.
The break-even chart uses tensor parallelism for multiple GPUs and assumes ideal bandwidth scaling; the planner adds a heuristic communication penalty, but neither replaces a topology-specific benchmark.
