Why we chose each component, and the assumptions and thresholds behind it — we publish our design decisions as they are.
Q.01
Why run Slurm + Kubernetes together?
Because the workloads are fundamentally different. HPC MPI / LLM training needs (1) resource allocation precisely matched to node count, (2) time-of-day queue policies, and (3) backfill — a fit for Slurm’s model. AI inference, notebooks, and services, by contrast, need (1) short lifetimes, (2) partial GPU sharing (MIG), and (3) auto-scaling — a fit for Kubernetes + Volcano. Time-slicing the same physical GPU pool (Slurm for overnight training / K8s for daytime services) raises utilization from around 30% to 70%+.
Q.02
Why does backfill perpetually defer short jobs?
The default backfill policy operates on a "large jobs first" basis, so short but low-priority jobs can fail to slot into a reservation window and wait indefinitely. Tuning Slurm’s `bf_max_job_user` (maximum jobs per user considered for backfill) and `bf_window` (backfill lookahead window) to the workload pattern resolves this. Standard values for research-institute sites: `bf_max_job_user=10`, `bf_window=2880` (2 days).
Q.03
How do you resolve inter-department resource disputes?
We apply three things together: FairShare policy + per-department Accounts + usage visibility from Luxe Vantage. FairShare adjusts priority based on the past N days of usage, so "the department that used the most gets lower priority on its next job" is applied automatically. When Vantage surfaces per-department hourly GPU usage in real time, self-alignment happens at the department-head level, and we observe disputes dropping 90% within 1–2 quarters.