Skip to content
Friendly disclaimer: flozi00 TechHub is a solo side-project next to a full-time job — personal learning notes, no official statements. Verify critical steps yourself.
flozi00 Logoflozi00 TechHub
ENDE

Scale-Up vs Scale-Out: Network Bandwidth and Collective Math

A bounded comparison of NVLink domains and inter-rack networks, with a ring all-reduce example and explicit assumptions.

2 min readflozi00
nvidianvlinkinfinibandethernetrocenetworkingllmhardware
Skip to the full article
Watch · explore · understand

A collective pays for the path its bytes actually take

Follow a ring all-reduce from placement through its two phases, keeping bandwidth units and assumptions consistent.

AI-assisted learning visualization · schematic article explanation

Step 1 / 4Choose the actual communication path

Scale-up connects GPUs locally; scale-out crosses system boundaries. Compare bytes per second with bytes per second, and one-way capacity with one-way capacity, accounting for shared ports and network cuts.

AlgorithmCurrent stepSelect a block to explain it
Local scale-up links
Nearby GPUs share a high-bandwidth local domain.
Scale-out network path
NICs, switches and shared links connect systems.
Place N GPUs and choose the path
Use sustained bandwidth in one consistent direction.
  1. Local scale-up links Place N GPUs and choose the path
    local path capacity
  2. Scale-out network path Place N GPUs and choose the path
    network cut and sharing
What to remember

Communication cost follows per-GPU bytes and achieved path bandwidth, not a rack-wide marketing total.

The ring formula is a bandwidth-only model. The article’s 900- and 100-GB/s examples are idealized inputs, not measured NVLink-versus-InfiniBand results; multiple links and contention can change the comparison.

Terms and the complete walkthrough
All-reduce
Combine every GPU’s contribution and return the result to all GPUs.
Sustained bandwidth
The transfer rate actually achieved over the relevant operation.
Network cut
The links connecting two groups whose aggregate capacity can limit traffic.
  1. Choose the actual communication path

    Scale-up connects GPUs locally; scale-out crosses system boundaries. Compare bytes per second with bytes per second, and one-way capacity with one-way capacity, accounting for shared ports and network cuts.

  2. Follow the two ring phases

    Reduce-scatter combines the GPUs’ contributions and leaves one result shard on each GPU. All-gather then distributes those shards so every GPU holds the complete reduced result.

  3. Count transferred bytes consistently

    The bandwidth-only ring term is 2(N−1)/N times V/B, with V contributed by each GPU. For eight GPUs the factor is 1.75; substitute achieved bandwidth for the selected path when modeling a real system.

  4. Compare the model with a measurement

    The term omits startup and contention, and NCCL can choose another algorithm. Measure the actual placement and message sizes before predicting the effect on a model workload.

All connections

  • Local scale-up links → Place N GPUs and choose the path: local path capacity
  • Scale-out network path → Place N GPUs and choose the path: network cut and sharing
  • Place N GPUs and choose the path → Reduce-scatter: start ring exchange
  • Reduce-scatter → All-gather: reduced shards
  • All-gather → Bandwidth term: 2(N−1)V / (NB): two-phase byte volume
  • Place N GPUs and choose the path → Bandwidth term: 2(N−1)V / (NB): N, V and sustained B
  • Bandwidth term: 2(N−1)V / (NB) → Check the real collective: ideal term to validate
Set your own pace. Pause at any time or select a step directly.

A multi-rack GPU cluster usually has a fast scale-up connection between nearby GPUs and a scale-out network between systems or racks. For GB200 NVL72, NVIDIA specifies a 72-GPU NVLink domain with 1.8 TB/s bidirectional per GPU and 130 TB/s aggregate across the rack. These are link-capacity figures, not the speed of a particular collective or application. NVIDIA's tuning guide gives 900 GB/s theoretical unidirectional NVLink bandwidth per GPU.

An 800 Gb/s XDR InfiniBand port carries a nominal 100 GB/s in one direction after converting bits to bytes. That is a 9:1 comparison of theoretical per-GPU NVLink capacity to one ideally dedicated 800 Gb/s port. It is not a universal 9× application speedup: a GPU may have multiple network links, a port may be shared, and topology, software and message size alter achieved bandwidth.

A ring all-reduce model

For a ring all-reduce over NN GPUs, with VV bytes contributed by each GPU and sustained per-GPU transfer bandwidth BB, the bandwidth-only time is

Tring≈2(N−1)NVB.T_{\mathrm{ring}} \approx \frac{2(N-1)}{N}\frac{V}{B}.

The factor covers the reduce-scatter and all-gather phases. NVIDIA's nccl-tests documentation uses this same 2(N−1)/N2(N-1)/N factor to relate all-reduce algorithm bandwidth to bus bandwidth. The expression omits startup latency, topology contention and overlap with computation; BB must be achieved bandwidth for the actual path, not an automatically assumed link rate.

As a deliberately idealized illustration, let N=8N=8 and V=2V=2 decimal GB per GPU. The ring factor is 1.75. At an assumed 900 GB/s, the bandwidth term is 1.75×2/900≈3.91.75 \times 2/900 \approx 3.9 ms. At an assumed 100 GB/s it is 3535 ms. These are lower-bound model calculations, not measured NVLink-versus-InfiniBand benchmark results. In a real cluster the path may combine NVLink and network hops, and NCCL may choose a different algorithm.

Which network limit matters?

For tensor parallelism, frequent activation collectives can make fast local GPU communication valuable. Data parallelism often exchanges gradients less frequently, but its communication can still dominate at large scale. Pipeline parallelism transfers activations between stages; expert parallelism adds all-to-all traffic whose load balance depends on expert routing. None of these patterns has a fixed “best fabric” independent of tensor sizes, placement and overlap with computation.

In scale-out design, check bandwidth available across the relevant network cut, not only individual port speed. Oversubscription, routing and congestion can reduce effective throughput. The 130 TB/s NVL72 figure is an aggregate of GPU link rates (72 × 1.8 TB/s ≈ 129.6 TB/s), not the bandwidth any one GPU or one collective receives. Compare like units and like scopes: per direction versus bidirectional, per GPU versus per rack, and nominal versus achieved.

For a purchasing or placement decision, measure the target workload with its actual parallelism layout and run a collective benchmark such as nccl-tests. Record system topology, GPU count, message sizes and achieved bandwidth. A datasheet ratio is a capacity clue, not a performance prediction.