Why we chose each component, and the assumptions and thresholds behind it — we publish our design decisions as they are.
Q.01
Why InfiniBand?
In large-scale MPI collectives (AllReduce, AllGather, etc.), a 1 μs difference in network latency translates to a 5–15% difference in total training/simulation time. InfiniBand XDR 800Gbps / NDR 400Gbps guarantees end-to-end latency around 200ns via an RDMA hardware path + adaptive routing, and — unlike RoCEv2 — requires no separate PFC/ECN tuning. Running the same workload on RoCEv2 leaf-spine ECMP adds routing-collision and congestion-control work, so IB is the more stable choice unless the operations team has sufficiently deep RDMA expertise.
Q.02
Why IBM Spectrum Scale?
Large-scale HPC workloads must handle three things well simultaneously: (1) metadata hot-spots, (2) failure-group (FG) isolation on disk failure, and (3) multi-site DR. Spectrum Scale 5.2 has been validated with distributed metadata NSDs + Spectrum Scale failure groups + AFM (Active File Management) caching together, giving it a clear edge in operational stability over Lustre for large writes. Lustre carries the risk of a single-node MDS bottleneck and tricky ldiskfs/ZFS backend tuning, which is dangerous at understaffed sites.
Q.03
Why Slurm?
The real-world usage patterns of research institutes, universities, and government supercomputing sites (per-department queue separation, priorities, backfill, fair-share) fit Slurm’s policy model better than PBS Pro or LSF. Slurm runs on an active community with a plugin architecture (SPANK), sacct accounting data, and burst-buffer integration, and it also integrates well at the usage-data (metrics) level with K8s-side schedulers such as Run:ai and Volcano.