01Challenge
The situation
Deciding between InfiniBand and RoCEv2 for a new cluster. RoCEv2 was 30% cheaper than IB, but NCCL efficiency was a concern.
CASEGaming / AI Subsidiary
TIER T2RoCEv2 400G cut costs 28% vs. IB — with no efficiency loss
Workload
Mixed AI Inference + Training
Scale
64 GPU nodes / RoCEv2 400Gbps
02Approach
The approach
- 014-week PoC on both fabrics with the same workload
- 02PFC/ECN/DCQCN tuning + adaptive routing
- 03Explicit RDMA via NCCL_NET_PLUGIN
- 04Side-by-side comparison report to justify the decision
03Results
Results
NCCL all-reduce efficiency
93% (vs IB 96%)
5-year TCO
−28%
Training step time
+3% (within tolerance)
Have a similar case?
Just tell us your workload type and scale, and we'll give you a first diagnosis within 30 minutes.
Request a consultation →