Why we chose each component, and the assumptions and thresholds behind it — we publish our design decisions as they are.
Q.01
What are the criteria for choosing between IB and RoCEv2?
The decision rests on three axes. (1) Workload latency sensitivity — if a 1 μs difference in MPI AllReduce translates directly to 1% performance, IB comes first. (2) Operations-team RDMA experience — RoCEv2 has many tuning parameters (PFC headroom, ECN threshold, DCQCN, etc.) and needs 1–2 quarters of stabilization on first adoption. (3) Cost — RoCEv2 is roughly 20–30% cheaper at the same bandwidth, but once operations staffing costs are included, IB often has the TCO advantage at large scale. A design mixing IB + Ethernet on the same fabric (separating storage and management networks) is also possible.
Q.02
How do you set the fat-tree blocking ratio?
It is determined by the share of collective traffic (AllReduce, etc.) that traverses the spine. AI training (60–80% NCCL collectives) requires non-blocking (1:1); HPC MPI can tolerate 1:2 (oversubscribed). Simple inference/serving workloads are fine down to 1:4. We validate with osu_bw and osu_alltoall measurements during the PoC phase.
Q.03
What are the criteria for adopting NDR (200/400 Gbps)?
To fully utilize a single GPU node’s PCIe Gen5 ×16 bandwidth (~64 GB/s, about 512 Gbps), 2× NDR 400Gbps NICs are recommended. For H100 / B300 × 8 GPU nodes, NDR 1 port = 400 Gbps is the minimum recommended spec. Expanding an existing EDR (100Gbps) site is also possible in stages via NDR down-converter mode.