Networking & Collective Communication

"A 10,000-GPU cluster is not 10,000 individual computers. It is a single parallel computer whose performance is determined not just by the GPUs, but by the bandwidth and latency of the wires between them." — Distributed Systems Architecture Principle

When 1,024 GPUs train a 70B model together, every single training step requires terabytes of gradient data to move between GPUs at microsecond latency. The network is not secondary infrastructure — it is a first-class performance component. A well-configured InfiniBand cluster achieves 85% scaling efficiency at 1,024 GPUs; a poorly configured one might achieve 40%. That 45-point gap translates directly to $3M per year in wasted compute on a 1,000-GPU cluster at $3/GPU-hr.

This chapter builds the complete picture from silicon interconnect to collective algorithm. By the end, you will be able to design the networking layer for a 10,000-GPU training cluster, tune NCCL for maximum throughput on both InfiniBand and RoCE fabrics, diagnose scaling efficiency regressions, and justify every topology and configuration decision quantitatively.

14.1 The Complete Communication Stack

Before optimizing anything, you need a mental model of the communication hierarchy. Every byte of gradient data, every activation tensor, every KV cache transfer in a distributed AI system passes through one of five communication levels — each with wildly different bandwidth, latency, and software API. The core insight that shapes every parallelism decision is this: optimize at the lowest possible level. NVLink is 18× faster than InfiniBand for the same bandwidth. This is not a minor optimization opportunity — it is the difference between Tensor Parallelism being viable or catastrophic.

14.2 NVLink and NVSwitch — Intra-Node Fabric

NVLink is NVIDIA's proprietary high-speed GPU-to-GPU interconnect. It is the single most important hardware feature enabling Tensor Parallelism in production. Without it, TP would require every AllReduce to cross InfiniBand — making the TP latency budget 18× worse and rendering TP at any scale infeasible for most models.

14.2.1 NVLink Architecture SHIPPED H100SHIPPED B200

── NVLink Physical Specifications ──────────────────────────────────────

NVLink 4.0 (H100 / H200):
  Links per GPU:          18 NVLink links
  Bandwidth per link:     50 GB/s bidirectional
  Total BW per GPU:       18 × 50 = 900 GB/s bidirectional
  Protocol:               NVIDIA proprietary (NOT PCIe)
  Connection topology:    GPU → NVSwitch ASIC (not direct GPU-GPU)
  Used for:               Tensor Parallelism, Tensor Parallel AllReduce

NVLink 5.0 (B200 / GB200):
  Bandwidth per link:     100 GB/s bidirectional
  Total BW per GPU:       18 × 100 = 1,800 GB/s = 1.8 TB/s
  Improvement:            2× over NVLink 4.0
  NVLink-C2C (GB200):     CPU-GPU unified memory at 900 GB/s

── NVLink vs PCIe Comparison ───────────────────────────────────────────

  PCIe Gen5 x16:          ~128 GB/s (one direction only)
  NVLink 4.0:             900 GB/s bidirectional (450 GB/s each way)
  NVLink speedup:         ~7× over PCIe for GPU-GPU transfers

  This 7× difference is why NVLink-C2C (Grace Blackwell) enables viable
  unified CPU-GPU memory: 900 GB/s CPU-GPU is only 7× slower than
  intra-GPU HBM access, making CPU DRAM a usable overflow tier.

14.2.2 NVSwitch — The Full-Bisection Fabric

NVSwitch is a dedicated ASIC that implements an all-to-all NVLink fabric inside a DGX/HGX node. The critical word is full bisection bandwidth: any GPU can communicate with any other GPU simultaneously at the maximum NVLink bandwidth. This is what makes AllReduce inside a node so fast — there is no contention.

14.2.3 NVLS — NVLink SHARP (In-Network AllReduce) SHIPPED H100

NVLS offloads AllReduce computation onto NVSwitch hardware, freeing GPU SM cycles. Without NVLS, the GPU's streaming multiprocessors must execute the reduction loop on every gradient element. With NVLS, the NVSwitch sums all values in dedicated combinatorial logic — the GPU SM touches zero cycles of the AllReduce.

── NVLS AllReduce vs Standard Software AllReduce ────────────────────────

WITHOUT NVLS (software ring on GPU SMs):
  Each GPU executes: result = my_grad + recv_grad  (N-1 times in ring)
  SM cycles consumed: 5–15% of total step time for AllReduce
  Latency (8 GPU intra-node): ~8–12 ms for 1B parameter gradient

WITH NVLS (NVSwitch hardware reduction):
  GPUs send shards to NVSwitch simultaneously
  NVSwitch dedicated logic sums all shards → broadcasts result
  GPU SM cycles consumed: ZERO (hardware does the math)
  Latency improvement: ~40% lower vs software ring
  Effective BW improvement: ~2× (no reduce overhead)

ENABLING NVLS:
  export NCCL_NVLS_ENABLE=1          # Enable (default on H100+)
  export NCCL_ALGO=NVLS              # Force NVLS (or leave auto)

  # Verify NVLS is active:
  NCCL_DEBUG=INFO python train.py 2>&1 | grep "NVLS"
  # Expect: "NCCL INFO Channel 00 : ... NVLS"

  NCCL 2.27 (2025) enhancement: Symmetric memory
  GPUs register identical virtual addresses → AllReduce uses
  single-copy zero-copy semantics → 7–9× latency reduction
  for small collectives (<256KB) — critical for TP per-layer AllReduce

14.3 InfiniBand — The Inter-Node Standard

When gradients must leave a node, they enter the inter-node fabric. InfiniBand (IB) is the dominant technology for AI training clusters in 2025–2026, used in NVIDIA DGX SuperPOD, most cloud provider HPC instances, and the majority of frontier model training runs. Its key properties — low latency (~1 μs), high bandwidth (400 Gbps per port), and native RDMA (Remote Direct Memory Access) — make it purpose-built for this workload.

14.3.1 InfiniBand Stack Architecture SHIPPED NDR 400G

── InfiniBand End-to-End Stack ──────────────────────────────────────────

PyTorch / NCCL / application
        ↓
NCCL library  (collective operations: AllReduce, AllGather, ReduceScatter)
        ↓
IB Verbs API  (libibverbs: queue pairs, completion queues, memory regions)
        ↓
RDMA kernel driver  (mlx5_core, ib_uverbs — Mellanox/NVIDIA kernel modules)
        ↓
Host Channel Adapter (HCA) — the GPU node NIC
  Mellanox ConnectX-7 (NDR 400G): most common H100 cluster NIC
  NVIDIA BlueField-3 DPU: NIC + ARM CPU + security in one package
        ↓
InfiniBand Switch Fabric
  Quantum2 (NDR 400G):   64 ports, 40 Tbps aggregate, SHARP-capable
  Quantum-X800 (XDR 800G): 2× bandwidth, 2025+ [ANNOUNCED]
        ↓
Remote HCA → Remote NCCL → Remote GPU HBM (via GPUDirect RDMA)

── NDR 400G Specifications (current standard, 2025–2026) ────────────────

  Port rate:              400 Gbps per port = 50 GB/s per direction
  Typical config:         8 × ConnectX-7 NICs per DGX node
  Per-node BW:            8 × 50 GB/s = 400 GB/s bidirectional
  Switch latency:         ~100 ns per hop
  End-to-end latency:     ~1 μs (2–3 hops in fat-tree)
  RDMA:                   YES — NIC writes directly to remote GPU HBM
  SHARP:                  YES — in-switch AllReduce computation

── IB Addressing ────────────────────────────────────────────────────────

  LID (Local Identifier):   16-bit address within subnet
  GID (Global Identifier):  128-bit address (subnet manager assigns)
  QP  (Queue Pair):         communication endpoint (TX QP + RX QP)
  Each NCCL channel = one IB Queue Pair

14.3.2 IB SHARP — In-Network AllReduce SHIPPED Quantum2

IB SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) performs AllReduce inside InfiniBand switches — the inter-node analog of NVLS. Without SHARP, gradient data traverses the network twice (scatter phase + gather phase). With SHARP, switches aggregate in dedicated compute units, cutting network traffic by 50% and latency by 30–40%.

── IB SHARP Mechanism ───────────────────────────────────────────────────

WITHOUT SHARP (traditional ring AllReduce):
  Phase 1 (ReduceScatter): GPU sends gradient chunks around the ring  →  crosses EVERY link
  Phase 2 (AllGather):     Aggregated result distributed back         →  crosses EVERY link again
  Total network traffic:   2 × gradient_size × (N-1)/N ≈ 2G per step

WITH SHARP (in-switch aggregation):
  GPUs send to nearest aggregation switch
  Switch compute units sum partial results
  Switch broadcasts final result downward
  Network traffic reduction: ~50%
  Latency reduction: ~30–40% (fewer hops)
  GPU SM cycles freed: SMs compute while SHARP runs

SHARP CONFIGURATION:
  export NCCL_SHARP_ENABLE=1
  export SHARP_COLL_LOCK_ON_COMM_INIT=1    # Pre-allocate resources at init
  export NCCL_SHARP_GROUP_SIZE_THRESH=16   # Use SHARP only for groups ≥ 16 ranks
  export NCCL_ALGO=SHARP                   # Or leave auto — NCCL selects for large msgs

SHARP LIMITATIONS:
  Only InfiniBand — not available on RoCEv2 Ethernet
  Requires SHARP daemon (sharpd) on every node
  Best for large-message AllReduce (>1MB)
  Small messages (<64KB) → use Tree algorithm instead

14.4 RoCEv2 — Ethernet-Based RDMA

Meta, Microsoft, Google, and other hyperscalers have built some of the world's largest AI training clusters on RoCEv2 (RDMA over Converged Ethernet) rather than InfiniBand. The technical reasons — and the specific engineering challenges Meta had to solve to make RoCE work at 24,576-GPU scale — are a masterclass in infrastructure tradeoffs.

14.4.1 IB vs RoCE Decision Framework

14.4.2 Making Ethernet Lossless: PFC and DCQCN

RDMA requires lossless transport — a single dropped packet causes retransmission that can stall an entire AllReduce ring, multiplying latency by 10×. Standard Ethernet drops packets whenever switch buffers overflow. Two complementary mechanisms prevent this:

── MECHANISM 1: Priority Flow Control (PFC) ─────────────────────────────

When switch output buffer reaches threshold:
  Switch sends PAUSE frame to upstream sender
  Sender halts transmission for PAUSE duration
  Result: No packet drops — lossless behavior

Problem: Head-of-Line Blocking (HOL)
  PAUSE blocks ALL flows in the same priority class
  Unrelated traffic (storage, management) pauses alongside RDMA
  Cascading PFC storms can dark out entire racks

Mitigation:
  Isolate RDMA on a dedicated priority queue (priority 3)
  mlnx_qos -i eth0 --pfc 0,0,0,1,0,0,0,0   # PFC on priority 3 only

── MECHANISM 2: DCQCN (Data Center Quantized Congestion Notification) ───

DCQCN prevents PFC storms by rate-limiting BEFORE buffers fill:

  Step 1: Switch detects congestion → marks packets with ECN
  Step 2: Receiver sends CNP (Congestion Notification Packet) to sender
  Step 3: Sender reduces transmission rate (CNPQ algorithm)
  Step 4: Rate recovers gradually when congestion clears
  Result: Buffer utilization stays low → PFC rarely triggers

Configuration (Mellanox ConnectX NIC):
  mlnx_qos -i eth0 --dscp2prio "set,106,3"    # DSCP 106 → priority 3
  cma_roce_tos=106   # in /etc/rdma/rdma.conf

14.4.3 Meta's CTS Protocol — Receiver-Driven Admission SHIPPED Meta RSC

Meta discovered that DCQCN reacts after congestion develops — at 400G speeds, switch buffers can fill in microseconds before DCQCN responds. Their solution, published at SIGCOMM 2024, is a receiver-driven admission protocol called CTS (Clear to Send) built into their NCCLX fork:

── Meta CTS Protocol (NCCLX fork) ──────────────────────────────────────

PROBLEM with standard DCQCN at 400G:
  ECN marking happens AFTER switch buffers start filling
  At 400G: buffers fill in ~1 μs before DCQCN reacts
  Brief but frequent drops → NCCL retransmissions → 3–5× slower

CTS SOLUTION — receiver-driven admission control:

  Phase 1 (SENDER requests):
    "I have N bytes to send to GPU_X"
    Sender WAITS — does NOT start transmitting immediately

  Phase 2 (RECEIVER grants):
    GPU_X checks available receive buffer space
    If space: sends 5-byte CTS packet back to sender

  Phase 3 (SENDER transmits):
    Only after CTS received does RDMA write begin
    Switch buffers NEVER overflow → PFC never triggers

  Result: Switch buffer utilization 10× lower than standard RoCE
  Zero packet drops under normal operation
  ~90% scaling efficiency at 24,576 GPUs (vs ~60% without CTS)

Implementation: NCCLX = Meta's NCCL fork with CTS
Used for: All Llama 3 and Llama 4 training runs
Published: SIGCOMM 2024 — "Reconfiguring Meta's production AI cluster"

14.5 Network Topology — Fat-Tree and Rail-Optimized Fabrics

Bandwidth numbers tell only half the story. The topology — how switches and cables are physically connected — determines whether that bandwidth is actually available to training workloads. The wrong topology can cut effective AllReduce throughput by 30–50% even on a 400G fabric.

── Rail-Optimized Topology: Why It Matters ──────────────────────────────

Without rail optimization (random NIC-to-switch assignment):
  GPU-0 on Node A sends via Rail0 → Spine → Rail2 → GPU-0 on Node B
  GPU-1's traffic also crosses same spine links → hotspot at spine
  Queue buildup → ECN → retransmissions → 30–50% throughput loss

With rail optimization (GPU-i → Rail-i always):
  GPU-0 (all nodes) → Rail0 ONLY → no cross-rail traffic
  Each rail carries exactly 1/N_rails of total AllReduce traffic
  Spine switches: balanced load, no hotspots
  Effective AllReduce bandwidth: full 400 GB/s utilized

NCCL configuration for rail-optimized topology:
  export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3  # GPU 0→mlx5_0, GPU 1→mlx5_1, etc.
  # CRITICAL: ordering must match GPU-to-NIC PCIe topology
  # Verify: NCCL_TOPO_DUMP_FILE=/tmp/topo.xml python -c "import torch.distributed"

14.6 NCCL — Complete Configuration Guide

NCCL (NVIDIA Collective Communication Library) is the software layer that abstracts all hardware differences — NVLink, InfiniBand, RoCE — into a unified collective API. Understanding NCCL is non-negotiable for distributed training performance. An unconfigured NCCL on a well-built InfiniBand cluster can achieve only 40–50% of available bandwidth. A properly configured NCCL consistently achieves 85–95%.

14.6.1 Collective Primitives Reference

14.6.2 Complete NCCL Production Environment Variables

── NCCL Production Configuration Reference ─────────────────────────────

# ── Debug and Diagnostics ──────────────────────────────────────────
export NCCL_DEBUG=WARN                  # WARN | INFO | TRACE
export NCCL_DEBUG_FILE=/tmp/nccl_%h_%p.log  # Per-host per-process log
# NCCL_DEBUG=INFO reveals: algorithm selection, topology detection, ring order

# ── Transport Selection ────────────────────────────────────────────
export NCCL_IB_DISABLE=0               # 0=enable IB/RoCE, 1=force TCP socket
export NCCL_IB_HCA=mlx5_0,mlx5_2,mlx5_4,mlx5_6  # Which HCAs (match to GPUs!)
export NCCL_SOCKET_IFNAME=eth0         # Fallback interface if IB unavailable

# ── InfiniBand / RoCE Performance ──────────────────────────────────
export NCCL_IB_CUDA_SUPPORT=1          # Enable GPUDirect RDMA — CRITICAL
export NCCL_NET_GDR_LEVEL=2            # 0=off, 1=same PCIe root, 2=system-wide
export NCCL_IB_GID_INDEX=3             # RoCE GID (check: ibv_devinfo -d mlx5_0)
export NCCL_IB_TC=106                  # DSCP for ECN marking on RoCE
export NCCL_IB_QPS_PER_CONNECTION=4   # QPs per NIC (1=default, 4=recommended 400G)

# ── Algorithm and Protocol ─────────────────────────────────────────
export NCCL_ALGO=Ring                  # Ring|Tree|CollNet|NVLS|PAT|auto
export NCCL_PROTO=Simple               # Simple|LL|LL128 (Simple=IB, LL128=NVLink small)

# ── Channels and Buffer ────────────────────────────────────────────
export NCCL_MIN_NCHANNELS=4
export NCCL_MAX_NCHANNELS=8            # More channels = more BW used
export NCCL_BUFFSIZE=4194304           # 4MB buffer/channel (larger=better for big msgs)

# ── SHARP (IB in-network) + NVLS (NVSwitch) ──────────────────────
export NCCL_SHARP_ENABLE=1             # IB SHARP for inter-node
export SHARP_COLL_LOCK_ON_COMM_INIT=1
export NCCL_NVLS_ENABLE=1             # NVLink SHARP for intra-node

# ── Timeout and Stability ──────────────────────────────────────────
export NCCL_TIMEOUT=300                # Watchdog timeout (seconds)
# Increase to 1800 for large-cluster checkpoint operations

NCCL 2.27 (2025) — Key new defaults:
  Symmetric memory: auto-enabled — 7-9× latency reduction for small collectives
  Direct NIC (Grace Blackwell): proxy threads eliminated → 10% step time reduction

14.6.3 GPUDirect RDMA — The Zero-Copy Path

GPUDirect RDMA allows the InfiniBand NIC to read from and write to GPU HBM directly, bypassing the CPU and host DRAM entirely. Without it, every NCCL send operation requires a GPU→CPU→NIC data hop, consuming 16 CPU cores and adding 2–3 μs latency. With it, data moves GPU HBM → NIC DMA queue → network in a single transaction.

── Data Path Comparison: With vs Without GPUDirect RDMA ─────────────────

WITHOUT GPUDirect (legacy):
  GPU HBM → [PCIe] → CPU copy buffer → [PCIe] → NIC DMA → network
  Extra latency:  +2–3 μs per collective message
  CPU overhead:   16 cores per DGX node (1 proxy thread per NIC × 8 + buffers)
  BW bottleneck:  Limited by PCIe Gen5 = 128 GB/s (NIC bandwidth underutilized)

WITH GPUDirect RDMA:
  GPU HBM → [PCIe P2P BAR mapping] → NIC DMA → network
  Latency:        ~0.5 μs (direct memory-mapped DMA)
  CPU overhead:   2–4 cores (metadata only, no data movement)
  Freed CPU cores: 12 per DGX node
  Performance:    10–20% improvement on communication-bound steps

VERIFICATION:
  nvidia-smi topo -p2p r          # Should show OK/PIX/NV#, not HOST
  NCCL_DEBUG=INFO python train.py 2>&1 | grep "GDR"
  # Expected: "NCCL INFO GDR enabled"
  
  # Benchmark NIC-to-GPU bandwidth:
  ib_write_bw -d mlx5_0 -x 3 --report_gbits      # server
  ib_write_bw -d mlx5_0 -x 3 <server> --report_gbits  # client
  # Target: ~380–395 Gbps (95%+ of 400G line rate)

14.7 Collective Communication Optimization

14.7.1 Compute-Communication Overlap

The highest-leverage networking optimization is not choosing the right algorithm — it is ensuring that communication never blocks computation. A training step where AllReduce runs sequentially after the backward pass wastes 100% of the communication time as dead GPU cycles. The goal is to make AllReduce invisible by overlapping it with the next computation.

── Overlap Strategy: Sequential vs Pipelined ────────────────────────────

WITHOUT OVERLAP (naive DDP):
  ┌────────────────────────────────────────────────────────────┐
  │  Forward (compute)  │  Backward (compute)  │  AllReduce   │
  └────────────────────────────────────────────────────────────┘
  Total: T_fwd + T_bwd + T_allreduce
  AllReduce fraction: typically 20–40% of step time = GPU dead time

WITH OVERLAP (DDP gradient bucketing):
  ┌────────────────────────────────────────────────────────────┐
  │  Forward Pass  │  Backward Pass (layer N)  │              │
  │                │  + AllReduce (bucket 0)    │              │
  │                │       + AllReduce (bucket 1)│             │
  └────────────────────────────────────────────────────────────┘
  Total: T_fwd + max(T_bwd, T_allreduce)
  Speedup: up to 1.3× for communication-heavy models

DDP CONFIGURATION for maximum overlap:
  model = torch.nn.parallel.DistributedDataParallel(
      model,
      bucket_cap_mb=25,           # Gradient bucket size MB (tune per model)
      gradient_as_bucket_view=True,  # Zero-copy gradient bucketing
      static_graph=True,          # Optimize for fixed computation graph
  )

FSDP CONFIGURATION for forward+backward overlap:
  model = FSDP(
      model,
      forward_prefetch=True,      # AllGather next layer while computing current
      backward_prefetch=BackwardPrefetch.BACKWARD_PRE,  # Prefetch during backward
      limit_all_gathers=True,     # Prevent OOM from too many inflight AllGathers
  )

14.7.2 Gradient Accumulation — Amortizing Communication

── Gradient Accumulation: Reducing AllReduce Frequency ──────────────────

WITHOUT accumulation (accum_steps=1, frequent communication):
  1000 training steps → 1000 AllReduce operations
  AllReduce fraction of time: ~25–40% (too high)

WITH accumulation (accum_steps=8, batched communication):
  125 "super-steps" → 125 AllReduce operations (8× fewer)
  AllReduce fraction of time: ~4–6%  (excellent)

Global batch size unchanged:
  micro_batch_per_gpu=4, accum_steps=8, world_size=512
  Global batch = 4 × 8 × 512 = 16,384 tokens per update

PyTorch FSDP implementation:
  for step, batch in enumerate(loader):
      ctx = model.no_sync() if (step % accum_steps != 0) else nullcontext()
      with ctx:
          loss = model(batch) / accum_steps  # Normalize gradient
          loss.backward()              # Grad accumulated, no AllReduce

      if (step + 1) % accum_steps == 0:
          optimizer.step()             # AllReduce fires HERE (once per 8 steps)
          optimizer.zero_grad()

14.8 Parallelism-to-Network Mapping

This is the section where networking and ML training architecture converge. Every parallelism strategy generates a specific communication pattern. The physical fabric it uses — NVLink or InfiniBand — determines whether that communication is an imperceptible overhead or a catastrophic bottleneck.

── Quantitative Justification: Why TP Must Stay on NVLink ───────────────

TP=8 AllReduce overhead per transformer layer:
  Message size per AllReduce: hidden_dim × seq_len × 2 bytes = 4096 × 2048 × 2 = 16 MB
  Number of AllReduces per 80-layer forward pass: 80 layers × 2 ops = 160 ops

  Over InfiniBand (IB) at 50 GB/s per NIC × 8 = 400 GB/s:
    Time per AllReduce = 2 × 7/8 × 16MB / 400GB/s = 0.07ms
    Total TP overhead = 160 × 0.07ms = 11.2ms per forward pass
    Wait — but NCCL ring uses multiple hops, and latency dominates small messages
    Realistic: 160 × 0.5ms = 80ms per forward pass (unusable)

  Over NVLink at 900 GB/s intra-node:
    Time per AllReduce ≈ 0.05ms (NVLS hardware, symmetric memory)
    Total TP overhead = 160 × 0.05ms = 8ms per forward pass (acceptable)

  IB vs NVLink for TP: 10× difference in overhead — not a minor optimization

14.9 Scaling Efficiency — Measuring Network Impact on MFU

Scaling efficiency is the ground truth metric for how well your cluster networking supports your training workload. It answers: "If I double the GPUs, how close to double do I get in throughput?" Every point of lost efficiency is wasted infrastructure cost. At 1,000 GPUs × $3/hr, going from 79% to 85% scaling efficiency saves ~$1.5M per year.

── Scaling Efficiency Definition and Measurement ────────────────────────

Scaling Efficiency (SE) = (Multi-node tokens/s) / (Single-node tokens/s × N_nodes)

Ideal SE = 100% — N GPUs deliver exactly N× single-GPU throughput
Typical SE = 70–90% for well-configured clusters at 1K+ GPUs

── LLaMA-3 70B Training Benchmark (empirical, 128-node × 8 GPU) ────────

  8 GPUs (1 node baseline):              4,800 tokens/s
  1,024 GPUs (128 nodes, 4D parallel):   488,000 tokens/s
  Scaling efficiency: 488,000 / (4,800 × 128) = 79.4%

  The 20.6% efficiency loss breaks down:
    AllReduce overhead (DP dimension):   ~8%   (gradient sync across 128 nodes)
    Pipeline bubbles (PP=8, 16 microbatch): ~6% (bubble = (PP-1)/(PP-1+M) = 7/23 = 30% → ~6% step time)
    Data loading stalls:                  ~3%   (storage I/O not fast enough)
    Other overhead:                       ~3%   (SLURM scheduling, kernel launch)

── How to Measure in Production ─────────────────────────────────────────

  # Step 1: Single-node baseline
  torchrun --nproc_per_node=8 train.py --steps 100 --profile
  # Record: tokens_per_second, avg_step_time_ms

  # Step 2: Multi-node run
  torchrun --nproc_per_node=8 --nnodes=128 train.py --steps 100 --profile
  # Compare: (multi-node tok/s) / (single-node tok/s × 128)

  # Step 3: Attribute the gap (via Nsight Systems or NCCL timing)
  nsys profile --trace=cuda,nvtx,nccl --output=profile python train.py
  # Look for: NCCL AllReduce rows serialized (bad) vs overlapped (good)

── Pipeline Bubble Analysis ──────────────────────────────────────────────

  Bubble fraction = (PP - 1) / (PP - 1 + num_micro_batches)

  PP=8, micro_batches=8:  bubble = 7/15 = 46.7% — terrible
  PP=8, micro_batches=16: bubble = 7/23 = 30.4% — still bad
  PP=8, micro_batches=32: bubble = 7/39 = 17.9% — acceptable
  PP=4, micro_batches=16: bubble = 3/19 = 15.8% — good

  Solution: Interleaved 1F1B scheduling with virtual pipeline stages
  Each physical PP stage contains V virtual stages → reduces bubble to 1/(V×PP)

Profiling & Diagnostic Cookbook

export NCCL_DEBUG=INFO
nsys profile --trace=cuda,nvtx,nccl -o profile_run python train.py --steps 20
# In nsys-ui: look at the "NCCL" row
# Metric: NCCL_total_time / total_step_time
# <10%: networking not the bottleneck
# 10–20%: acceptable, minor tuning possible
# 20–30%: investigate topology and overlap
# >30%: networking IS the primary bottleneck

# NCCL bandwidth test (from nccl-tests package)
mpirun -np 1024 --hostfile hosts all_reduce_perf   -b 1M -e 4G -f 2 -g 1 -c 1
# Output: busBW (GB/s) vs message size
# Compare against theoretical peak: (N-1)/N × NIC_BW × N_NICs
# If achieved < 80% of theoretical: topology or config problem

ibstat mlx5_0 | grep -E "State|Physical|Rate"
# Expect: Active / Polling / 400 Gb/sec

perfquery -x 3              # Check port error counters
# PortXmitDiscards > 0 → congestion, bad cable, or config
# PortRcvErrors > 0    → signal integrity issue (check cable)

ib_write_bw -d mlx5_0 --report_gbits [server]
# Target: ~380–395 Gbps (95%+ of 400G)
# If < 350 Gbps: GPUDirect disabled, PFC misconfigured, or cable issue

IF AllReduce > 30% of step time:
  → Check if NCCL operations are overlapped with compute (nsys)
  → If NOT overlapped: add DDP(static_graph=True, gradient_as_bucket_view=True)
  → Increase gradient_accumulation_steps to reduce AllReduce frequency
  → Check NCCL_IB_CUDA_SUPPORT=1 (GPUDirect RDMA enabled)

IF raw bandwidth test < 80% of peak:
  → Check NCCL_IB_HCA ordering (must match GPU-to-NIC PCIe topology)
  → Verify rail-optimized topology: GPU-i traffic → NIC-i → Rail-i
  → Check oversubscription ratio (spine bandwidth adequate?)
  → Enable NCCL_IB_QPS_PER_CONNECTION=4

IF ibstat shows errors or link speed != 400G:
  → Replace cable (most common cause)
  → Check SFP/QSFP transceiver compatibility
  → Verify PFC and DCQCN configuration on all switches

IF scaling efficiency degrades beyond 1024 GPUs:
  → Consider IB SHARP (export NCCL_SHARP_ENABLE=1)
  → Evaluate gradient accumulation to reduce sync frequency
  → Profile per-GPU step times — identify straggler nodes (thermal throttle?)

Worked Example

DP groups:    1,024 GPUs / (TP=8 × PP=4) = 32 DP replicas
Model size:   70B parameters × 2 bytes/param (BF16) = 140 GB
Gradient size per step: 140 GB (full model gradient in DP AllReduce)
# But FSDP ReduceScatter sends only 1/DP fraction per node, then AllGather
Message per GPU per ReduceScatter: 140 GB / 32 = 4.375 GB
Total AllReduce volume: 2 × (31/32) × 140 GB = 271 GB of network traffic

Available IB bandwidth (one DP group = all 128 nodes):
  Per-node NIC BW: 8 NICs × 50 GB/s = 400 GB/s
  Ring algorithm: bandwidth = (N-1)/N × per_node_BW = (31/32) × 400 GB/s = 387 GB/s

AllReduce time = message_size / bandwidth = 271 GB / 387 GB/s = 0.70 seconds

With IB SHARP (50% traffic reduction):
  Effective time = 0.70 × 0.5 = 0.35 seconds

LLaMA-3 70B forward+backward compute time per step (H100, BF16):
  ~2.5–3.5 seconds at global batch = 4M tokens

AllReduce fraction WITHOUT overlap or accumulation:
  0.70 / 3.0 = 23%  TOO HIGH — exceeds 10% budget

With overlap (DDP/FSDP gradient bucketing, 60% overlap achievable):
  Effective overhead = 0.70 × (1 - 0.60) = 0.28s → 9.3%  within budget

With SHARP + overlap:
  Effective overhead = 0.35 × (1 - 0.60) = 0.14s → 4.7%  excellent

Target: AllReduce fraction < 10% without relying on overlap
AllReduce time per sync: 0.70s (without SHARP), 0.35s (with SHARP)

Compute time per accumulated super-step with accum=4:
  super_step_time = 4 × 3.0s = 12.0s
  AllReduce fraction = 0.70 / 12.0 = 5.8% — easily within budget

Recommendation:
  Minimum: NCCL_SHARP_ENABLE=1 + DDP gradient bucketing
  Conservative: gradient_accumulation_steps=4 + SHARP
  Both: AllReduce cost becomes negligible (<5% of step)
  Global batch with accum=4: 4M × 4 = 16M tokens per update
  (verify convergence at this batch size before committing)