The Starting Point
A GPU driver update is an unavoidable operational event. A CUDA 12.x → 12.y update means synchronously reprovisioning all 200 nodes. The old procedure was:
1. Document the changed spec (half a day) 2. Redeploy + verify one PoC node (1 day) 3. Redeploy in batches of 50 (4 days) 4. Verification + rollback scenario review (1 day)
That's 6.5 days gone. Meanwhile, large jobs are paused or migrated to another node group.
What Orbit Solved
1. Catalog-based image management
An image definition like `Image[ID=os-cuda12y-2026-02]` is registered in the catalog. The image is cached on the PXE server and mapped to node group policy.
2. Node group policy
The 200 nodes are partitioned into groups like `group:gpu-train-01`. A one-line group policy change applies `image=os-cuda12y-2026-02`.
3. Management server HA + failover
Redundant management servers. Even if one dies, failover takes 30 seconds. This prevents a single management-server failure from stalling a 200-node redeploy.
4. cloud-init + ansible post-processing
Right after OS boot, cloud-init configures SSH keys and hostname. Then an ansible playbook installs the NVIDIA driver, CUDA, and NCCL. Ordering is guaranteed.
Results
``` ops$ orbit deploy --group gpu-train-01 --image os-cuda12y-2026-02 [orbit] PXE caching... done (45s) [orbit] Initiating 200 nodes... [orbit] All nodes booted (200/200) in 7m42s [orbit] cloud-init complete (200/200) in 4m20s ✓ Total: 12m02s ```
What's Left for the Operator
The operator only runs one command plus verification. The change history accumulates in the catalog automatically. Rollback is a one-line command with the previous image ID.
Case
/case-studies/orbit-200-node-zerotouch-bootstrap — a 200-node cluster at an enterprise R&D organization.