Scale-Up vs Scale-Out: The Networking Math Behind LLM Clusters

Every LLM cluster is two networks stacked on top of each other: a scale-up fabric that makes a rack behave like one large GPU (NVLink + NVSwitch), and a scale-out fabric that stitches thousands of those racks together (InfiniBand or Ethernet/RoCE). Vendors sell both with peak bandwidth numbers; the interesting question is what those gigabits buy you under real collectives. This article works the math explicitly — link rates, ring all-reduce cost, speedup ceilings — and then looks at the silicon reasons the two tiers exist at all.

1. The Link Math

Raw numbers first, pinned to current-generation datasheets. NVLink 5 (Blackwell, GB200/GB300 NVL72) gives every GPU 18 links at 100 GB/s per direction — 900 GB/s per direction, 1.8 TB/s bidirectional — routed over a copper backplane to nine NVSwitch trays that form a single non-blocking 72-GPU domain with 130 TB/s aggregate bidirectional bandwidth¹². InfiniBand's current generations are NDR at 400 Gb/s per port (Quantum-2) and XDR at 800 Gb/s per port (Quantum-3 / Quantum-X800, 144 × 800 Gb/s per 4U switch, 115.2 Tb/s switching capacity)³⁴. Ethernet sits at 400G/800G with RoCEv2, driven by the Spectrum-X platform (SN5600: 64 × 800G OSFP, 51.2 Tb/s) and ConnectX-8 SuperNICs that expose 2 × 400 Gb/s per GPU on GB300-class systems⁵².

The per-GPU row is the one that matters. A B200 inside an NVL72 domain has 900 GB/s of switch-fabric bandwidth to any other GPU in the rack. The same GPU reaching another rack through scale-out goes over at most 800 Gb/s (100 GB/s) of NIC bandwidth — 9× less, and that is the best case (1:1 NIC-to-GPU provisioning with no oversubscription). Watch the unit trap here: 900 vs 800 looks like near-parity until you convert GB/s against Gb/s. Notice the units too: vendor sheets alternate between "1.8 TB/s" (bidirectional sum) and "900 GB/s" (per direction); the collective math below only works with per-direction numbers.

2. The Latency Hierarchy (and Why It Matters Less Than Bandwidth)

Measured and datasheet latency ranges for the two tiers:

• Intra-domain (NVLink/NVSwitch): GPU-to-GPU across an NVSwitch hop lands around 0.1–0.25 µs; published measurements on NVSwitch fabrics report roughly one hundred nanoseconds of switch latency plus a fraction of a microsecond of NIC-side overhead⁶.
• NIC + one switch hop (NDR InfiniBand): small-message RDMA round-trip on a single switch is roughly 1–2 µs; the Quantum-2 switch itself contributes on the order of 100 ns per hop in cut-through mode⁶⁴.
• RoCEv2 over Ethernet: typically 1.5–2.5 µs same-switch RTT, with Ethernet switch hops at roughly 400–600 ns — several times InfiniBand per hop, but the same order of magnitude⁶.

The gap between 0.2 µs and 2 µs looks like a 10× problem. It mostly isn't, for LLM collectives. A latency bound applies per message:

where $\alpha$ is the per-message latency and $S$ the chunk size. NCCL pipelines rings with chunks of 512 KB and larger⁷, so the latency term amortizes to a handful of round-trips. The bandwidth term, by contrast, scales with the payload itself:

For a 2 GB all-reduce even a 10 µs latency penalty is noise against milliseconds of transfer time. Latency dominates only small collectives — MoE small-message all-to-all, barrier-style metadata sync, decode-time inference. Bandwidth dominates everything that moves activations or gradients. That is the whole reason NVLink domains exist: they buy a 9× bandwidth improvement, not primarily a latency one.

3. Collective Math: Why Communication Caps Scaling

The ring all-reduce cost model

A ring all-reduce over $N$ GPUs with per-GPU payload $V$ splits into a reduce-scatter (data circulates once, $V$ per GPU reduced to $V/N$ per GPU) and an all-gather (the reduced shards circulate once). Each phase moves $\frac{N-1}{N}V$ per GPU, so total time is:

where $B$ is the per-GPU bus bandwidth (per direction — in a ring every GPU sends and receives simultaneously). The factor $\frac{2(N-1)}{N}$ approaches 2 and never exceeds it — this is the famous "algorithmic bandwidth" property: a ring stays bandwidth-bound no matter how many GPUs you add. The N-factor, computed:

Worked example: tensor-parallel attention activation all-reduce

Take one transformer layer with a tensor-parallel attention output all-reduce, the classic TP communication hot spot: hidden size 16,384, batch 8, sequence 8,192, BF16:

Ring all-reduce time at three fabrics, computed with the formula above:

Now put compute next to it. The GEMM behind this activation is a 16k→64k expansion, about $1.4 \cdot 10^{14}$ FLOPs; at a B200's dense BF16 rate of about 2.25 PFLOPS per GPU, compute time is 31.3 ms on 2 GPUs but only 7.8 ms on 8 and 0.87 ms on 72. Adding the communication term gives the actual scaling picture:

Two conclusions fall straight out of the arithmetic. Going from 2 to 8 GPUs inside an NVLink domain yields 2.81× instead of the ideal 4× — the communication term eats about 35%. Going from 8 to 72 GPUs yields only 2.15× instead of 9× — at 84% communication fraction, you are spending more time moving activations than matmul-ing them, and that with NVLink-class bandwidth. This is the ceiling no marketing slide removes: the speedup saturates at

and since $T_c$ shrinks with $1/N$ while the communication term stays roughly constant, every strong-scaling workload hits the wall at the $N$ where $\frac{2(N-1)}{N}\frac{V}{B} \approx T_c/1$.

Trees and in-network reduction

Tree algorithms reach the same $\mathcal{O}(V \log N)$ data volume but win on latency for small $V$: a binary-tree all-reduce is $\log N$ steps instead of $N-1$ steps, at the cost of $\frac{2(N-1)}{N}$-style bandwidth being harder to achieve concurrently at the root. The escape from software trees is in-network reduction. On InfiniBand, SHARP performs the reduction in the switch ASIC itself as data streams through a hardware reduction tree — the SHARP Streaming-Aggregation paper reports MPI_Allreduce at about 95% of raw network bandwidth and 2–5× the bandwidth of host-based reduction at medium to large message sizes⁸, and up to 5.1× lower MPI_Allreduce latency at 7,861-node scale on TACC Frontera⁹. NVIDIA reports SHARPv2 (HDR generation) doubling all-reduce bandwidth, worth 17% end-to-end BERT training throughput in their MLPerf v1.0 submission (NVIDIA-reported)¹⁰. The NVLink domain has its own equivalent: NVSwitch ASICs implement SHARP-style multicast and reduction for the intra-rack collectives², and NVIDIA claims 4× bandwidth efficiency with FP8 SHARP on the NVLink Switch chip of GB200 NVL72 (vendor-claimed)². In-network reduction does not break the bandwidth bound — it raises the achievable fraction of it and removes one traversal.

4. The Silicon View: Why Scale-Out Costs More per GB/s

If Ethernet were free, everyone would just use it everywhere. The reason NVLink domains exist at all is per-bit physics.

Copper vs SerDes. Inside the rack, NVLink 5 runs 100 GB/s links over passive copper through the NVL72 backplane — a fraction of a GB in reach (hence rack-scale), but near-zero transceiver energy. Outside the rack, every bit crosses PAM4 SerDes at 100–200 Gb/s per lane. The IEEE 802.3df task-force survey quantified 200 Gb/s-per-lane SerDes at roughly 3.5–4.5+ pJ/bit projected depending on channel loss, versus long-reach 100G/lane SerDes measured at 4.6–6.5 pJ/bit; only intra-package XSR links reach ≈1.7 pJ/bit¹¹. That is per SerDes end; each pluggable link pays both ends.

Optics. Beyond ~2–3 m of passive copper (or ~7 m with an active copper cable at 2–4 pJ/bit), scale-out hops to optics: 800G pluggable modules burn 15–20 pJ/bit all-in (25–30 W per 1.6T module with DSP-dominated power), with the DSP alone at 6–7 W per 800G module in the 100G/200G-lane generation¹¹. Co-packaged optics and silicon photonics promise to cut this toward 3–5 pJ/bit, but that is roadmap math, not shipping-rack math (as of 2026).

The arithmetic of one hop: moving 800 Gb/s over an optical run costs roughly $800 \cdot 10^9 \cdot 15 \cdot 10^{-12} \approx 12$ W per direction per link end, versus fractions of a watt for a copper NVLink lane inside the rack. Multiply by the tens of thousands of links in a 100k-GPU cluster and scale-out interconnect becomes a measurable — sometimes dominant — fraction of the rack's power budget. Copper's attenuation sets the inverse constraint: retimers and DSPs buy reach at pJ/bit costs that climb with loss, which is why the NVLink domain is a rack, not a row: GB300 NVL72's "one domain" is nine copper-backplane-connected NVSwitch trays filling one rack, not a happily arbitrary design choice². Scale-up stays where copper is cheap; scale-out begins where copper stops working.

5. Topology: Bisection Bandwidth and Oversubscription

Scale-out fabrics are fat-trees. The key metric is bisection bandwidth: cut the cluster in half and sum the link bandwidth crossing the cut. In an ideal (non-blocking) 2-level fat-tree, bisection bandwidth equals the host-facing bandwidth — every GPU can talk to every other at line rate for any traffic pattern. That requires the spine layer to have exactly the aggregate bandwidth of the leaves; in practice, cost pushes operators to oversubscribe: e.g., 2:1 or 4:1 (for every 4 Gb/s of leaf downlink, 1 Gb/s of uplink). Under a uniform-injection pattern, a 2:1-oversubscribed fabric halves the worst-case effective all-reduce bandwidth even though every port still says "800 Gb/s" on the label. NVIDIA's AI-factory reference architecture for GB300 builds a two-level SN5600 leaf-Spine fabric with 144 × 400 Gb/s uplinks per 72-GPU rack at 1:1 — deliberately non-blocking, per-plane — precisely to keep eth math honest for all-reduce².

Multiplane designs split each SuperNIC across 2–8 independent leaf-spine planes and load-balance packets across them in hardware, relaxing radix limits at constant bisection bandwidth⁵. Their cost is fragmenting any single collective's paths; NCCL must be multiplane-aware.

What does "GB300 NVL72 is one domain" mean for an all-reduce? A 72-GPU ring can be drawn entirely inside the copper domain, so the ring runs at 900 GB/s per GPU. The same 72-GPU all-reduce spread over nine 8-GPU servers connected by Ethernet must carry $\frac{2 \cdot 71}{72}$ of the payload through inter-node links at whatever per-GPU NIC bandwidth remained unoversubscribed; even at 1:1 XDR provisioning, that's the 5.29 ms vs 4.71 ms row plus one to two switch hops — and the whole collective now lives or dies with the fabric's oversubscription ratio, hot-path congestion control, and adaptive routing.

6. Decision Table by Workload

For MoE, the numbers say: at any realistic token flow — say 65,536 tokens × 4,096 hidden in BF16, 0.54 GB per GPU per dispatch — an 800-Gb/s per-GPU Ethernet/NIC allocation moves it in about 5.37 ms, while the same dispatch inside an NVLink domain's effective ~450 GB/s half-duplex share takes about 1.19 ms. But MoE dispatch saturates below these wire-rate numbers in practice, because all-to-all traffic is the pattern most sensitive to oversubscription and incast (many GPUs hitting one leaf switch simultaneously at the combine step): the fabric's lossless behavior and adaptive routing give out before the link rate does. What saturates first is not raw Gb/s — it is the effective bisection bandwidth under the actual permutation, plus the small-message latency floor for expert-routing metadata.

Rules of thumb: TP ≤ domain width (8 on HGX, 72 on NVL72); PP is fabric-agnostic; DP is a scale-out problem every fabric handles if bandwidth is provisioned 1:1; MoE all-to-all wants NVLink-class bandwidth inside the expert groups and a non-oversubscribed scale-out fabric, with SHARP/multiplane offload wherever the fan-out leaves the rack.

7. Critical View: Marketing Numbers vs Usable Bandwidth

The datasheet number is not the number your collective gets. Three systematic gaps:

1. Peak Gb/s is per-direction at 100% utilization. Real RoCEv2 fabrics sustain 60–90% of line rate under collective traffic depending on congestion control tuning; InfiniBand with SHARP is measured at about ≈95% of network bandwidth⁸. The gap multiplies with every hop.
2. "Aggregate" bandwidth is a sum, not a fabric property. The NVL72's celebrated 130 TB/s is $\sum 72 \cdot 1.8$ TB/s bidirectional — the sum of all GPUs' link bandwidth, not a number any single collective sees¹. A ring all-reduce on NVL72 achieves an algorithmic bandwidth of $\frac{N}{2(N-1)} \cdot 900 \approx 456$ GB/s — outstanding, but "130 TB/s" it is not, and a marketing slide comparing "130 TB/s NVLink vs 51.2 Tb/s Ethernet switch" is comparing a rack-wide aggregated sum to a per-switch capacity.
3. Bisection bandwidth and oversubscription are rarely printed. Computed example: a "100,000-GPU fabric" built from 800 Gb/s links at 4:1 spine oversubscription has a worst-case bisection of $\frac{1}{4} \cdot N \cdot 100$ GB/s — 2.5 PB/s, which sounds enormous until you divide per-GPU: 25 GB/s usable worst-case per GPU instead of the marketed 100 GB/s. Under a bandwidth-bound ring all-reduce, the communication term simply runs at $B = 25$ GB/s, quadrupling $T_{\text{ring}}$ relative to the datasheet claim. Unified 4×-per-generation "AI fabric" claims inherit all three gaps at once.

The honest comparison, per-GPU and per-direction: NVLink 5 gives 900 GB/s in-domain; XDR and 800G RoCEv2 provision up to 100 GB/s at the ideal 1:1 ratio; oversubscription and protocol efficiency scale that number downward, never upward. Scale-up buys a ~4.5–9× bandwidth multiple over a well-provisioned scale-out link, and the collective math says that multiple is exactly what TP-heavy and MoE workloads spend. Everything else — DP gradients, PP handoffs, checkpoint and storage traffic — runs fine on the scale-out fabric, which is why huge training clusters exist at all.

Related Resources

• NVIDIA NVLink Solutions
• NVIDIA GB200 NVL72 overview