The Symptom
Measuring osu_allreduce during the PoC, we saw about 70 GB/s across 64 H100 nodes. On a single NDR 200 Gbps port, that was roughly 60% of the 95% efficiency (about 116 GB/s) that sm-aware NCCL should deliver. Training wall-clock ran 35% longer than the spec assumption, and that much GPU time was being wasted.
The Hypotheses
We suspected three things at once:
1. Topology — in the fat-tree, ECMP repeatedly picked the same spine, causing collisions. 2. Message size — automatic ring-vs-tree algorithm selection went wrong for large messages. 3. CPU/NUMA — NCCL proxy threads pinned to the wrong NUMA node.
Measurement
- `NCCL_DEBUG=INFO` + `NCCL_PROTO=Simple` comparison — automatic protocol selection was accurate. - `numactl --hardware` + nccl-tests affinity options — NUMA was fine. - Mellanox `ibdiagnet` + enabling adaptive routing — this was the crux.
The Fix
We applied three changes:
1. Rail-Optimized cabling — distribute the 8 NICs of an 8-GPU node 1:1 across 8 leaf switches. 2. Enable adaptive routing — `AR` mode + `SHARP` (SHARPv3 IB collective acceleration). 3. Explicit NCCL_IB_HCA — specify the 8 HCAs explicitly so the library selects the correct NICs.
Results
| Measurement | Before | After | |------|--------|-------| | osu_allreduce 64MB (64 nodes) | 70 GB/s | 112 GB/s | | Efficiency vs. spec | 60% | 96% | | Training step time | 1.42s | 0.94s | | 64-node wall-clock (1 epoch) | 8.4 hours | 5.5 hours |
What We Left for Operations
- A `runbook` for adaptive routing / SHARP configuration - Automated nccl-tests (run via Slurm prolog, results surfaced in Vision) - A regression measurement script for firmware changes
The data from this PoC also fed into RFP authoring — you can state objective acceptance criteria in an RFP, such as "Acceptance standard: NCCL all-reduce 64MB across 64 nodes at 112 GB/s or higher."